You can read about the utility here.

Automatic Memory Management makes use of the SHMFS – a pseudo file system just like /proc. Every file created in the SHMFS is created in memory.

Unfortunately using MEMORY_TARGET or MEMORY_MAX_SIZE together with Huge Pages is not supported. You have to choose either Automatic Memory Management or HugePages. In this post i´d like to discuss AMM and Huge Pages.

Automatic Memory Management (AMM)

AMM – what it is

Automatic Memory Management was introduced with Oracle 11g Release 1 and automated sizing and re-sizing of SGA and PGA. With AMM activated there are two parameters of interest:

• MEMORY_TARGET
• MEMORY_MAX_TARGET

MEMORY_TARGET specifies the oracle system-wide usable amount of memory while MEMORY_MAX_TARGET specifies the upper bound of which the DBA can se MEMORY_TARGET to. If MEMORY_MAX_TARGET is not specified it defaults to MEMORY_TARGET.

For AMM to work there is one important requirement: Your system needs to support memory mapped files (on Linux typically mounted on /dev/shm).

According to the documentation the following platforms support AMM:

• Linux
• Solaris
• Windows
• HP-UX
• AIX

More information on AMM can be found here, here, here and here.

Advantages

• SGA and PGA automatically adjusted
• dynamically resizeable
• Not swappable

Disadvantages

• Only available on a limited number of plattforms

Bug for Feature?

• Does not work together with HugePages – so it is either AMM or HugePages

Before deciding lets see what HugePage are:

HugePages

HugePages – what are they?

Here is one description:

Hugepages is a mechanism that allows the Linux kernel to utilise the multiple page size capabilities of modern hardware architectures. Linux uses pages as the basic unit of memory, where physical memory is partitioned and accessed using the basic page unit. The default page size is 4096 Bytes in the x86 [and x86_64 as well; note by Ronny Egner] architecture. Hugepages allows large amounts of memory to be utilized with a reduced overhead. Linux uses “Transaction Lookaside Buffers” (TLB) in the CPU architecture. These buffers contain mappings of virtual memory to actual physical memory addresses. So utilising a huge amount of physical memory with the default page size consumes the TLB and adds processing overhead. The Linux kernel is able to set aside a portion of physical memory to be able be addressed using a larger page size. Since the page size is higher, there will be less overhead managing the pages with the TLB.

(Source: http://unixfoo.blogspot.com/2007/10/hugepages.html)

Advantages

• Huge Pages are not swappable; thus keeping your SGA locked in memory
• Overall memory performance is increased: Since there are less pages to scan the memory performance is increased
• kswapd needs far less resources: kswapd regularly scans the page table for infrequent accessed pages which are a candidate for paging to disk. If the page table is large kswapd will use a lot of resources in therm of CPU. With Huge Pages enabled the page table is much smaller and Huge Pages are not subject to swap so kswapd will use less resources.
• Improves TLB hit ratio due to less entries thus increasing memory performance further. The TLB is a small cache on cpu which stores virtual to physical memory mappings

Disadvantages

• Should be allocated at startup (allocating huge pages at runtime is possible but will fail probably due to memory fragmentation; so it is advisable to allocate them at startup)
• dynamically allocating hugepages is buggy

Linux memory management with and without Huge Pages

The following two figures try to illustrate how memory access with and without huge pages work. As you can see every process in a virtual memory operating system has it´s own process page table which points to a system page table.For oracle processes running on linux it is not uncommon to use the same physical memory regions due to accessing the SGA or the block cache. This is illustrated in the figures below for the pages 2 and 3; they are access by both processes.

Without huge pages memory is divided in chunks (called: “pages”) of 4 KB on Intel x86 and Intel x86_64 (the actual size depends on the hardware platform). Operating system offering virtual memory (as most modern operating system do, for instance linux, solaris, hp-ux, aix and even windows) present each process a continuous addressable memory (“virtual memory”) which consists of memory pages which reside in memory or even on disk (“swap”). See here for more information.

The following file taken from the wikipedia article mentioned above illustrates the concept of virtual memory:

From the process’ view it looks like it is solely running on the operating system. But it is not. In fact there are a lot of other processes running.

As mentioned above memory is presented to processes in chunks of 4 kb – a so called “page”. The operating system manages a list of pages – the “page table” – for each process and for the operating system as well which maps the virtual memory to physical memory.

The page table can be seen as “memory for managing memory”.Each page table entry (PTE) takes:

• 4 bytes of memory per page (4 kb) per process on 32-bit intel and
• 8 byte of memory per page (4 kb) per process on 64-bit intel

For more information see my post and the answer on the Linux Kernel Mailing List (LKML).

So for a process to touch every page on a 64-bit system with 16 GB memory there are required:

• for the memory referenced by the process: 4.2 million PTE  (~ 16 GB) with 8 byte each = 32 MB
• PLUS for the system page table: 4.2 million PTE (~ 16 GB) with 8 byte each = 32 MB
• equals to 64 MB for the whole page table as counted in /proc/meminfo [PageTables]
  

On systems running oracle databases with a huge buffer cache and highly active processes (= sessions) it is not uncommon for the processes to reference the whole SGA and parts of the PGA after a while. Taken the example above assuming a buffer cache of 16 GB this adds up to 32 MB per process for the page table. 100 processes will consume 3.2 GB! Thats a lot of memory which is not available and solely used to manage memory.

The size of the page table can be querien on linux as follows:

cat /proc/meminfo | grep PageT
PageTables:      25096 kB

This command show the size of the Page Table (sum of system and all process page tables). This amount of memory is unusable for all processes and solely for managing the memory. On system with a lot of memory and a huge sga/pga and many dedicated server connections the page table can be several GByte in size!

The solution for this memory wastage is to implement huge pages. Huge pages increase the memory chunks from 4 kb to 2 MB so a page table entry still takes 8 bytes in 64-bit intel but references 2 mb – thats more efficenty by a factor of 512!

So taken our example from above with a buffer cache of 16 GB (16384 MB) referenced completely by a process the page table for the process will be:

• 16384 GB referenced / 2 MB per page = 8192 PTE with each 8 byte needed = 65536 Byte or 65 KB

I guess the advantage is obvious: 32 MB with no huge pages vs. 65 KB with huge pages!

Is my system already using huge pages?

You can check this by doing:

cat /proc/meminfo | grep Huge

There are three possibilities:

No huge pages configured at all

cat /proc/meminfo | grep Huge
HugePages_Total:  0
HugePages_Free:   0
HugePages_Rsvd:   0
Hugepagesize:     2048 kB

Huge pages configured but not used

cat /proc/meminfo | grep Huge
HugePages_Total:  3000
HugePages_Free:   3000
HugePages_Rsvd:   0
Hugepagesize:     2048 kB

Huge pages configured and used

[root@rac1 ~]# cat /proc/meminfo | grep Huge
HugePages_Total:  3000
HugePages_Free:   2601
HugePages_Rsvd:   2290
Hugepagesize:     2048 kB

How to configure huge pages?

1. edit /etc/sysctl.conf and add the following line:

vm.nr_hugepages = <number>

Note: This parameter specifies the number of huge pages. To get the total size you have to multiply “nr_hugepages” by “cat /proc/meminfo | grep Hugepagesize”. For 64-bit Linux on x86_64 the size of one Huge Page is 2 MB. So for a total amount of 2 GB or roughly 2000 MB you need 1000 Pages.

2. edit /etc/security/limits.conf and add the following lines:

<oracle user>    soft    memlock    unlimited
<oracle user>    hard    memlock    unlimited

3. Reboot the server and check

Some real world examples

The following are two database systems not using Huge Pages. Lets see how much memory is spend just for managing the memory:

System A

The following is an example of a Linux based database server running two database instances with approx. 8 GB SGA in total. At the time of sampling there are 444 dedicated server sessions connected.

MemTotal:     16387608 kB
MemFree:        105176 kB
Buffers:         21032 kB
Cached:        9575340 kB
SwapCached:       1036 kB
Active:       11977268 kB
Inactive:      2378928 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:     16387608 kB
LowFree:        105176 kB
SwapTotal:     8393952 kB
SwapFree:      8247912 kB
Dirty:            9584 kB
Writeback:           0 kB
AnonPages:     4754720 kB
Mapped:        7130088 kB
Slab:           256088 kB
CommitLimit:  16587756 kB
Committed_AS: 22134904 kB
PageTables:    1591860 kB
VmallocTotal: 34359738367 kB
VmallocUsed:      9680 kB
VmallocChunk: 34359728499 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
Hugepagesize:     2048 kB

As you notice approx. 10% of all available memory is used for the Page Tables.

System B

System B is a Linux based system with 128 GB memory running one single database instance with 34 GB SGA and approx 400 sessions:

MemTotal:     132102884 kB
MemFree:        596308 kB
Buffers:        472620 kB
Cached:       111858096 kB
SwapCached:     138652 kB
Active:       65182984 kB
Inactive:     53195396 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:     132102884 kB
LowFree:        596308 kB
SwapTotal:     8393952 kB
SwapFree:      8112828 kB
Dirty:             568 kB
Writeback:           0 kB
AnonPages:     5901940 kB
Mapped:       33971664 kB
Slab:           915092 kB
CommitLimit:  74445392 kB
Committed_AS: 48640652 kB
PageTables:   12023792 kB
VmallocTotal: 34359738367 kB
VmallocUsed:    279912 kB
VmallocChunk: 34359456747 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
Hugepagesize:     2048 kB

In this example Page Tables allocate 12 GB of memory. Thats a lot of memory.

Some laboratory examples

Test case no. 1

The first test case is quite simple. I started 100 dedicated database connections with a delay of 1 second between each database connection. Each session will log on and sleep for 200 seconds and log off. In the operating system i will monitor the page table size with increasing and decreasing database sessions.

foo.sql script

exec dbms_lock.sleep(200);
exit

doit.sql script

doit.sql
for i in {1..100}
do
echo $i
sqlplus system/manager@ora11p @foo.sql &
sleep 1
done

Results without Huge Pages

The following output were observed without huge pages:

while true; do cat /proc/meminfo | grep PageTable && sleep 3; done

PageTables:      58836 kB
PageTables:      60792 kB
PageTables:      62808 kB
PageTables:      64808 kB
PageTables:      66560 kB
PageTables:      68780 kB
PageTables:      70084 kB
PageTables:      72044 kB
PageTables:      72296 kB
PageTables:      74184 kB
PageTables:      76804 kB
PageTables:      79000 kB
PageTables:      80928 kB
PageTables:      82932 kB
PageTables:      84652 kB
PageTables:      86576 kB
PageTables:      88936 kB
PageTables:      90896 kB
PageTables:      94120 kB
PageTables:      96424 kB
PageTables:      98212 kB
PageTables:     100304 kB
PageTables:     101868 kB
PageTables:     103960 kB
PageTables:     105996 kB
PageTables:     108108 kB
PageTables:     109992 kB
PageTables:     111404 kB
PageTables:     113584 kB
PageTables:     114860 kB
PageTables:     116856 kB
PageTables:     118276 kB
PageTables:     120256 kB
PageTables:     120316 kB
PageTables:     120240 kB
PageTables:     120616 kB
PageTables:     120316 kB
PageTables:     121456 kB
PageTables:     121480 kB
PageTables:     121484 kB
PageTables:     121480 kB
PageTables:     121408 kB
PageTables:     121404 kB
PageTables:     121484 kB
PageTables:     121632 kB
PageTables:     121484 kB
PageTables:     121480 kB
PageTables:     120316 kB
PageTables:     120320 kB
PageTables:     120316 kB
PageTables:     120320 kB
PageTables:     121460 kB
PageTables:     121652 kB         <==== PEAK AROUND HERE
PageTables:     121500 kB
PageTables:     121540 kB
PageTables:     120096 kB
PageTables:     118136 kB
PageTables:     116188 kB
PageTables:     114192 kB
PageTables:     112236 kB
PageTables:     110240 kB
PageTables:     106556 kB
PageTables:     103792 kB
PageTables:     101820 kB
PageTables:      97916 kB
PageTables:      95900 kB
PageTables:      95120 kB
PageTables:      93104 kB
PageTables:      91848 kB
PageTables:      89852 kB
PageTables:      87860 kB
PageTables:      85896 kB
PageTables:      83868 kB
PageTables:      81940 kB
PageTables:      79944 kB
[...]

Results with Huge Pages

while true; do cat /proc/meminfo | grep PageTable && sleep 3; done

PageTables:      27112 kB
PageTables:      27236 kB
PageTables:      27280 kB
PageTables:      27320 kB
PageTables:      27344 kB
PageTables:      27368 kB
PageTables:      27396 kB
PageTables:      27416 kB
PageTables:      31028 kB
PageTables:      31412 kB
PageTables:      37668 kB
PageTables:      37912 kB
PageTables:      39964 kB
PageTables:      39756 kB
PageTables:      39740 kB
PageTables:      41312 kB
PageTables:      41436 kB
PageTables:      41508 kB
PageTables:      42192 kB
PageTables:      42196 kB
PageTables:      42528 kB
PageTables:      43036 kB
PageTables:      43232 kB
PageTables:      45616 kB
PageTables:      44852 kB
PageTables:      44540 kB
PageTables:      44552 kB
PageTables:      44728 kB
PageTables:      44748 kB
PageTables:      44764 kB
PageTables:      45936 kB
PageTables:      46992 kB
PageTables:      48128 kB
PageTables:      49264 kB
PageTables:      50312 kB
PageTables:      51056 kB
PageTables:      52244 kB
PageTables:      53496 kB
PageTables:      54256 kB
PageTables:      55296 kB
PageTables:      56440 kB
PageTables:      57712 kB
PageTables:      58240 kB
PageTables:      58824 kB
PageTables:      59612 kB
PageTables:      60656 kB
PageTables:      62468 kB
PageTables:      63592 kB
PageTables:      64700 kB
PageTables:      65820 kB
PageTables:      66916 kB
PageTables:      68344 kB
PageTables:      69144 kB
PageTables:      70260 kB
PageTables:      71044 kB
PageTables:      72172 kB
PageTables:      73224 kB
PageTables:      73684 kB
PageTables:      74736 kB
PageTables:      75828 kB
PageTables:      76952 kB
PageTables:      78068 kB
PageTables:      79180 kB
PageTables:      78604 kB
PageTables:      79384 kB
PageTables:      79384 kB
PageTables:      80064 kB
PageTables:      80092 kB
PageTables:      80096 kB
PageTables:      80096 kB
PageTables:      80096 kB
PageTables:      80084 kB
PageTables:      80096 kB
PageTables:      80092 kB
PageTables:      80096 kB        <=== PEAK AROUND HERE
PageTables:      80096 kB
PageTables:      79408 kB
PageTables:      79400 kB
PageTables:      79400 kB
PageTables:      79396 kB
PageTables:      79392 kB
PageTables:      79392 kB
PageTables:      79392 kB
PageTables:      79392 kB
PageTables:      79392 kB
PageTables:      79396 kB
PageTables:      79396 kB
PageTables:      70260 kB
[...]

Observations

• without huge pages page table size peaked at approx. 120 MB
• with huge page page table size peaked at approx. 80 MB

In this simple test case using huge pages used only 66% of memory.

Because of the choosen test case memory saving is not that big because we did not referenced that much memory from the SGA in our processes. Tests would be much clearer with larger parts (e.g. buffer cache) of the SGA referenced.

Conclusion

HugePages offers some important advantages over AMM, for instance:

• minimizing cpu-cycles used for scanning memory pages which are candidates for swapping thus freeing cpu-cycles for your database,
• minimizing memory spend for managing memory references

The latter point is the most important one. Especially systems with large memory amounts dedicated to SGA and PGA and many database sessions (> 100) will benefit from using Huge Pages. The more memory dedicated to SGA and PGA and the more sessions connected with the database the larger the memory savings from using Huge Pages will be.

From my point of view even if AMM simplifies memory management by including both PGA and SGA the memory (and cpu) savings from using Huge Pages are more important than just simlifying memory management.

So if you have an SGA larger than 16 GB and more than 100 sessions using Huge Pages is definetly worth trying. On system with only a few sessions using Huge Pages will give some benefit as well but only by reduding cpu-cycles needed for scanning the memory pages.

RMAN Compression Type	Compression Algorithm used	Adv. Compression License required?	Backup set size	CPU Load
BASIC	BZIP2 (100k record size?)	No	small	medium to high
NONE	none	No	largest; approx. db size	extremely small
LOW	LZO	YES	somewhat smaller than using NONE	low
MEDIUM	ZLIB	YES	medium	medium
HIGH	BZIP2 (900k record size?)	YES	smallest	highest

This article is intended to take a look at the different compression methods available in Oracle 11g and to compare them.

Test environment

The environment being used was a freshly created 11g Release 2 database with some smaller tables in it. The total sum of all segments equals to 4.88 GB. All database data files excluding the temporary ones are 7.3 GB total. Excluding temporary and undo data files total size equates to 5.9 GB.

The server was running on VMWARE with one dedicated CPU core per host and 4 GB memory.

Configure RMAN

To configure RMAN to use compression at all you can use:

CONFIGURE DEVICE TYPE DISK BACKUP TYPE TO COMPRESSED BACKUPSET;

or

CONFIGURE DEVICE TYPE TAPE BACKUP TYPE TO COMPRESSED BACKUPSET;

To configure the different backup algorithms:

CONFIGURE COMPRESSION ALGORITHM 'BASIC';
CONFIGURE COMPRESSION ALGORITHM 'NONE';
CONFIGURE COMPRESSION ALGORITHM 'LOW';
CONFIGURE COMPRESSION ALGORITHM 'MEDIUM';
CONFIGURE COMPRESSION ALGORITHM 'HIGH';

Test results

The following table displays the test results for the chosen test environment:

	NONE (=uncompressed)	BASIC	LOW	MEDIUM	HIGH
Size	4080192 KB	356924 KB	430200 KB	373288 KB	238200 KB
CPU Load	< 5% per Channel	70 to 80% per Channel	20% per Channel	50% per Channel	100% per Channel
Test 1	281 sec	291 sec	213 sec	203 sec	2992 sec
Test 2	334 sec	267 sec	187 sec	189 sec	long, very long
Test 3	272 sec	278 sec	189 sec	212 sec	long, very long
Test 4	324 sec	299 sec	157 sec	188 sec	long, very long
Test 5	260 sec	257 sec	157 sec	192 sec	long, very long
Minumum	260	257	157	188	n/a
Maximum	334	299	213	212	n/a
Avg.	294	278	180	196	n/a

As you can see from the table HIGH compression does an incredibly high load on the machine and take extremely long but produces the smallest backup set size. I suppose the long backup time might be explained by the fact that the environment is running under VMWARE which adds additional overhead and slows down CPU. Surprisingly BASIC compression (which is available without advanced compression license) does a good job as well and produces the second smallest backup set but takes nearly as long as doing uncompressed backups. But in other environment with faster CPUs this will change!

In the test environment used either LOW or MEDIUM compression seems to be the best choice. Due to the fact MEDIUM produces a approx. 15% smaller backup set but taking only a few seconds more time to complete i would rank MEDIUM on 1st and LOW on second.

Conclusion

RMAN offers backup compression since version 10. But with 11g Oracle added several new compression algorithms which lets you adjust the IO- and CPU-consumption and the resulting backup set size by choosing between five algorithms: NONE, DEFAULT, LOW, MEDIUM, HIGH. The stronger the compression the smaller the backup size but the more CPU-intensive the backup is. You basically change CPU-cycles for IOPs. If you do not have the advanced compression license BASIC compression will produce reasonable compression rates at moderate Load. If you have the licence you have a lot more options to suit your needs.

Regardless of the results in THIS test case YOUR results will be different due to: different data which leads to different compression ratios, different storage architecture for data files and backup destination (local disks, SAN, NAS, Tape), different CPU resources (faster/slower CPUs), rman parallelism). The test performed here are intended to give you some basic idea how backup time and backup size will change depending on the compression chosen.

!! Because of many factors influencing backup speed and backup size you have to test backup and backup compression in your environment yourself. There is no general rule of thumb. !!

If you want to test and optimize your rman backup you basically have three major switches to play with:

• compression algorithmn,
• rman parallelism and
• data transfer mechanism (SAN or Ethernet [this includes: iSCSI, NFS, CIFS, Backup to tape over Ethernet])

In addition to that i add some parameters recommended by myself.

Note that for running Oracle Grid Infrastrucure (aka “Clusterware”) some additional parameters might be required.

Values required by Oracle according to the documentation

The following tables list the required MINIMUM values according to the oracle documentation.

Parameter name in /etc/sysctl.conf	Description	MINIMUM value
kernel.sem	"semmsl semmns semopm semmni"
kernel.sem:semmsl	total semaphores for each set	250
kernel.sem:semmns	total number of semaphores systemwide	32000
kernel.sem:semopm	Maximum number of operations for semop call	100
kernel.sem:semmni	total semaphore sets	128
kernel.shmall	total number of shared memory pages that can be used system wide	2097152
kernel.shmmax	the maximum size of a single shared memory segment in bytes that a linux process can allocate	Either 4 GB - 1 byte, or half the size of physical memory (in bytes), whichever is lower. Default: 536870912
kernel.shmmni	system wide number of shared memory segments	4096
fs.file-max	maximum number of open files system-wide	6815744
fs.aio-max-nr	concurrent outstanding i/o requests	1048576
net.ip_local_port_range	minimum and maximum ports for use	Minimum: 9000 Maximum: 65500
net.rmem_default	the default size in bytes of the receive buffer	262144
net.rmem_max	the maximum size in bytes of the receive buffer	4194304
net.wmem_default	the default size in bytes of the send buffer	262144
net.wmem_max	the maximum size in bytes of the send buffer	1048576

.

Shell Limits for Oracle User in /etc/security/limit.conf	Description	Hard Limit
nofile	number of open files	65536
nproc	number of processes	16384
stack	stack size in kb	10240

/dev/shm file system	largest value of MEMORY_MAX_TARGET or MEMORY_TARGET of all instances

Calculate the required values

The following table shows calculation formulas for setting the required kernel parameters for running oracle 11g release 2 on Linux.

Parameter name in /etc/sysctl.conf	Description	MINIMUM value	Calculation
kernel.sem	"semmsl semmns semopm semmni"
kernel.sem:semmsl	total semaphores for each set	256	set fixed to 256
kernel.sem:semmns	total number of semaphores systemwide	32000	2 * sum (process parameters of all database instances on the system) + overhead for background processes + system and other application requirements
kernel.sem:semopm	Maximum number of operations for semop call	100	set fixed to 100
kernel.sem:semmni	total semaphore sets	128	semmns divided by semmsl, rounded UP to nearest multiple to 1024
kernel.shmall	total number of shared memory pages that can be used system wide	2097152	ceil(shmmax/PAGE_SIZE). (PAGE_SIZE is usually 4096 bytes unless Big/Huge Pages used) get PAGESIZE by doing: "getconf PAGESIZE"
kernel.shmmax	the maximum size of a single shared memory segment in bytes that a linux process can allocate	Either 4 GB - 1 byte, or half the size of physical memory (in bytes), whichever is lower. Default: 536870912	largest value of SGA_MAX_SIZE, SGA_TARGET,MEMORY_MAX_SIZE or MEMORY_TARGET of all instances
kernel.shmmni	system wide number of shared memory segments	4096	set fixed to 4096...did not find any formula
fs.file-max	maximum number of open files system-wide	6815744	>= 6815744 (dont know why this large....but should be enough for most databases)
fs.aio-max-nr	concurrent outstanding i/o requests	1048576	no adjustment; set fix to 1048576
net.ip_local_port_range	minimum and maximum ports for use	Minimum: 9000 Maximum: 65500	no adjustment, fixed
net.rmem_default	the default size in bytes of the receive buffer	262144	>= 262144
net.rmem_max	the maximum size in bytes of the receive buffer	4194304	>= 4194304
net.wmem_default	the default size in bytes of the send buffer	262144	>= 262144
net.wmem_max	the maximum size in bytes of the send buffer	1048576	>= 1048576

The default page size needed for some calculations can be obtained by doing:

getconf PAGESIZE

The output states the default (not huge page size) in bytes.

The shell limits shown above should be enough initially. However if you have many data files (> 1000) i would increase the number of open files by doubling the value. All other parameters are sized big enough for almost every environment.

For using the new Automatic Memory Management Feature which automatically sized SGA and PGA Oracle uses a pseudo file system /dev/shm. If activating this feature by setting MEMORY_MAX_SIZE or MEMORY_TARGET in your init.ora you have to size /dev/shm appropriately. The following table outlines how to do it:

/dev/shm file system	largest value of MEMORY_MAX_TARGET or MEMORY_TARGET of all instances

You can make this change permanent by editing your /etc/fstab. For instance the following change will set /dev/shm to a size of 13 GB:

...
shmfs /dev/shm   tmpfs   size=13g 0
...

Addition values recommended by Ronny Egner

Note: The values mentioned there are values set from experience. Set them at your own risk!

Value	Description	Recommended Value
vm.swappiness ( in /etc/sysctl.conf)	adjusts the swap affinity. Range from 0 to 100. Lower values mean pages remain longer in memory before swapping them out to disk. Larger values swap out infrequently used memory pages faster. The default value is “60″.	5

At the moment the following tests are planned and will be published during the next days:

• suddenly turning off the power and restarting the node
• terminating private network connect between the cluster nodes
• recovering an ACFS file system which will not mount automatically
  (was not able to reproduce more than once; sorry guys)
• overwriting the ASM disk header with the disk group being offline
• corrupting an online and active ASM disk by writing chunks of random data to the disk randomly
• simulating disk errors by removing the device from the operating system
• corrupting  the OCR
• corrupting the Voting Disk

The environment used for the posts are explained in detail here.

Useful scripts can be found here.

After overwriting the ASM disk header while the disk group was offline we will now put some load on the full running cluster and corrupt the asm disk slightly.

Testcase

Put load on the database

Prior starting this test i created a larger table with one column and approx 1.2 GB size and an empty table with the same structure. So for our simple test we will just copy one table into another table. This forces oracle to read the database blocks and do some writes as well.

create table test2_empty as select * from test where rownum<1;

insert into test2_empty select * from test;

Corrupting the disk

According to fdisk the asm lun has a size of 21474836480 bytes:

Disk /dev/sde: 21.4 GB, 21474836480 bytes
64 heads, 32 sectors/track, 20480 cylinders
Units = cylinders of 2048 * 512 = 1048576 bytes

 Device Boot      Start         End      Blocks     Id  System
/dev/sde1               1       20480    20971504   83  Linux

And the disk is indeed the mirror partner of the disk we destroyed and re-added in the earlier part.

[root@rac1 ~]# oracleasm querydisk /dev/sde1
Device "/dev/sde1" is marked an ASM disk with the label "DISK003B"

We will corrupt the disk by writing 512 byte chunks of random data from /dev/urandom to 1000 different locations all over the lun. Given the lun size this makes 41943040 blocks of 512 bytes each.

Our command to corrupt the lun will be:

dd if=/dev/random bs=512 count=1 of=/dev/sde1 seek=nnnn

Where “seek=nnn” is a number from 0 to 41943040-1.

Note that this time we manipulate the second mirror disk of disk group DATA2 – DISK003B. In test no 3 we overwrote the asm disk header of DISK003A and re-added the disk to the data group. In this example we manipulate DISK003B. If there were undetected errors rebalancing the disks we will discover it now.

For this we first of all needed a small little random number generator within the limits of 0 to 41943040; here it is:

#!/bin/bash

function random()
{
 od -d /dev/urandom | sed -e 's/^[0-9]* //' -e 's/ //g' |\
 while read L ; do echo -n $L ; done |\
 dd bs=1 count=${1:-10} 2>/dev/null
}

A=$(random 10) #return random number with 10 numbers
B=$(expr $C % 41943040)  # keep number within range
echo $B

With this random number generator we created a small shell script for corrupting the asm lun which looks like this:

dd if=/dev/urandom of=/dev/sde1 bs=512 count=1 seek=986024
dd if=/dev/urandom of=/dev/sde1 bs=512 count=1 seek=1594423
dd if=/dev/urandom of=/dev/sde1 bs=512 count=1 seek=2236024
[..]

Expected test results

So what are we expecting?

According to oracle documents (note 416046.1) there are two different scenarios:

• If the primary extent is detected to be corrupt (asm only reads from primary extent unless a read preference set) ASM retries the read. If re-read data is also corrupt ASM fails over to the secondary extent. If data in secondary extent is valid data in primary extent will be overwritten with the good data from the secondary extent.
• If data in secondary extent is corrupt it will remain undetected. If new data is written the new data will be written both to the primary and secondary disk. This will most likely overwrite the corrupted data. But if the primary extent fails oracle will read the secondary extent and will discover the corruption as well -  but this time the corruption is not fixable. You will end up with restoring/recovering the corrupt database blocks and probably much more.

In addition to that i expect ASM and/or the database to automatically fix detected corruptions.

Testing

Message no. 1: Database detected and fixed corruption

During our Insert statement running the database alert.log showed:

Fri Oct 02 10:47:21 2009
Hex dump of (file 6, block 156029) in trace file /u01/app/oracle/diag/rdbms
  /ora11p/ora11p1/trace/ora11p1_ora_30023.trc
Corrupt block relative dba: 0x0182617d (file 6, block 156029)
Bad header found during multiblock buffer read
Data in bad block:
 type: 180 format: 3 rdba: 0xb19a95b2
 last change scn: 0x6f92.6c357e5d seq: 0x1 flg: 0x5a
 spare1: 0xfd spare2: 0x80 spare3: 0x4f3c
 consistency value in tail: 0x1591148c
 check value in block header: 0x2abf
 block checksum disabled
Reading datafile '+DATA2/ora11p/users02.dbf' for corruption at rdba:
  0x0182617d (file 6, block 156029)
Read datafile mirror 'DISK003B' (file 6, block 156029) found same corrupt data
Read datafile mirror 'DISK003A' (file 6, block 156029) found valid data
Repaired corruption at (file 6, block 156029)

Message no. 2: Database detected corruption

Corrupt block relative dba: 0x01c12d35 (file 7, block 77109)
Bad check value found during validation
Data in bad block:
 type: 2 format: 2 rdba: 0x01c12d35
 last change scn: 0x0000.00253bd5 seq: 0x73 flg: 0x04
 spare1: 0x0 spare2: 0x0 spare3: 0x0
 consistency value in tail: 0x3bd50273
 check value in block header: 0x94cb
 computed block checksum: 0xf0eb
Trying mirror side DISK003B.
Reread of blocknum=77109, file=+DATA2/ora11p/datafile/undotbs01_02.dbf. found
 same corrupt data
Reread of blocknum=77109, file=+DATA2/ora11p/datafile/undotbs01_02.dbf. found
 valid data

Note: This time the database does not say anything about the corruption being fixed.

Message no. 3: ASM detected and fixed corruption

Mon Oct 05 09:32:15 2009
WARNNING: cache read a corrupted block group=DATA2 fn=1
 blk=893 from disk 0
NOTE: a corrupted block from group DATA2 was dumped to /u01/app/oracle/diag/asm
 /+asm/+ASM1/trace/+ASM1_ora_4739.trc
WARNNING: cache read(retry) a corrupted block group=DATA2 fn=1
 blk=893 from disk 0
NOTE: cache repaired a corrupted block group=DATA2 fn=1 blk=893
 from disk 0

Conclusion

First of all: regardless of the caveats described below the insert statement most of the time completed successfully:

SQL> insert into test2 select * from sys.test;
134217728 rows created.

Second: During our tests ASM performed well and stable. We were consistently able to work on the disk groups even if the disk group missed their mirrors.

However i discovered some problems as well:

• ASM has no ability to check data on disk groups completely (i.e. compare data on all mirror partners and check for corruptions); so corruptions on secondary extents will remain undetected until primary extent fails and secondary extent is read
• ASM nor the database does always fix corruptions found in primary extents (see message 2 above… corruption was detected but NOT corrected and remained on disk) as indicated by note 416046
• Checking database with RMAN and “backup validate database” finds and reports errors but does not fix ANY error found. The error message no. 2 originated from a “select * from test….” but rman produces this type of error messages in alert.log as well. From the Metalink document which describes handling of block corruptions in ASM i would have expected ASM or the database to fix these errors automatically.
• There are scenarios where corrupted primary blocks prevent the database from starting at all. During our tests we have seen this two times with different error messages. One sample error message stack is reproduced in error message no. 3 coming from the asm instance and in the following lines below coming from the database instance:

Exception [type: SIGILL, Illegal operand] [ADDR:0x105DC01]
[PC:0x105DC01, kkestRCSBase()+221] [flags: 0x0, count: 1]
Errors in file /u01/app/oracle/diag/rdbms/ora11p/ora11p1/
trace/ora11p1_ora_5206.trc  (incident=48254):
ORA-07445: exception encountered: core dump [kkestRCSBase()+221]
[SIGILL] [ADDR:0x105DC01] [PC:0x105DC01] [Illegal operand] []
Incident details in: /u01/app/oracle/diag/rdbms/ora11p/ora11p1/
incident/incdir_48254/ora11p1_ora_5206_i48254.trc
Mon Oct 05 09:33:47 2009
Trace dumping is performing id=[cdmp_20091005093347]
Mon Oct 05 09:33:49 2009
Exception [type: SIGILL, Illegal operand] [ADDR:0x105DC01]
[PC:0x105DC01, kkestRCSBase()+221] [flags: 0x0, count: 1]
Errors in file /u01/app/oracle/diag/rdbms/ora11p/ora11p1/
trace/ora11p1_ora_5296.trc  (incident=48310):
ORA-07445: exception encountered: core dump [kkestRCSBase()+221]
[SIGILL] [ADDR:0x105DC01] [PC:0x105DC01] [Illegal operand] []
Incident details in: /u01/app/oracle/diag/rdbms/ora11p/ora11p1/
incident/incdir_48310/ora11p1_ora_5296_i48310.trc

Although we met this errors we were able to fix them: In one case dropping the disk containing the corrupted blocks was enough to start the instance again. The second incident reproduced above required the corrupted disk to be dropped, re-added and ASM instance to be restarted before we were able to successfully start the database instance.

• Dropping a corrupted disk and re-adding the disk requires the disk group the disk belonged to be taken offline. Otherwise labeling the former member disk failed with “device or resource busy”. I will investigate this further.

Summarizing my experiences until now i would go with ASM and normal or even high redundancy. This enables far more options for fixing errors without having to rebuild the disk group and restoring everything from tape/disk.

DBFS – An Overview

Since the introduction of LOBs the database was used to store all kinds of files, for instance images, audio files, video files or even text files. Starting with Oracle 11g Release 1 introduced SecureFiles which introduced compression and de-duplication.

Oracle Database Filesystem (DBFS) creates a mountable file system which can be used to access files stored in the database as SecureFile LOBs. DBFS is a shared file system like NFS and consists of a server (the database) and a client. Just like a traditional database connection server and client can be on different systems which communicate over SQLNet. The same applies to DBFS as well.

The following figure illustrates DBFS:

(Source: http://download.oracle.com/docs/cd/E11882_01/appdev.112/e10645/adlob_fs.htm)

Install DBFS

Requirements

The following requirements must be met to use DBFS:

• only available for 32-bit and 64-bit Linux
• installed FUSE package
• installed kernel development package
• installed oracle client libraries

Install FUSE

• Download FUSE in Version 2.7.3 from http://fuse.sourceforge.net
  (Note: I initially tested with FUSE 2.8.1 but was not able to get it working; so stick with 2.7.3 or 2.7.4)
• Install kernel development packages matching your kernel (to query your kernel type “uname -a”)
• Compile and install FUSE:
  • tar -xvfz fuse-2.7.3.tar.gz
  • cd fuse-2.7.3
  • ./configure –prefix=/usr –with-kernel=/usr/src/kernels/`uname -r`-`uname -p`
  • make
  • make install
  • depmod
  • modprobe fuse
  • chmod 666 /dev/fused
  • echo “/sbin/modprobe fuse” >> /etc/rc.modules
  • ldconfig

Create DBFS File System

Creating a user to store the file system in

Prior creating a DBFS File system you must create a user for storing the file system in. DBFS file systems can be created for every user. The role DBFS_ROLE must be granted to users having dbfs file system, for instance:

create user dbfs identified by dbfs default tablespace
users quota unlimited on users;

grant dbfs_role to dbfs;
grant create session to dbfs;
grant create procedure to dbfs;
grant create table ro dbfs;
alter user dbfs default role all;

Create a simple file system

To create the DBFS file system execute the script “dbfs_create_filesystem.sql” being connected as user to store the file system in. The syntax is:

sqlplus <username>/<password>@<database>
@$ORACLE_HOME/rdbms/admin/dbfs_create_filesystem.sql <tablespace name>
<file system name>

To create a file system for user dbfs created earlier and storing the content in tablespace USERS and naming the file system TESTDBFS:

sqlplus dbfs/dbfs@ora11p @$ORACLE_HOME/rdbms/admin/dbfs_create_filesystem.sql
USERS TESTDBFS

dbfs_create_filesystem.sql creates a partitioned file system by creating multiple physical segments in the database and distributing files randomly in it. In cases when the files are extremely big and make a big percentage of the total file system this can cause an error “ENOSPC” to appear even if the file system is not full. In such cases the script “dbfs_create_filesystem_advances.sql” can be used to create a non-partitioned file system.

Create an advanced file system

Creating a standard file system does not use de-duplication or compression. To use this features you have to create the file system using the script “dbfs_create_filesystem_advanced.sql”. The calling syntax is:

sqlplus <username>/<password>@<database>
@$ORACLE_HOME/rdbms/admin/dbfs_create_filesystem_advanced.sql
<tablespace name> <file system name>
<compress-high | compress-medium | compress-low | nocompress>
<deduplicate | nodeduplicate>
<encrypt | noencrypt>
<partition | non-partition>

To create a de-duplicated, highly compressed, not encrypted and partitioned file system for user dbfs2 stored in tablespace USERS:

sqlplus dbfs2/dbfs2@ora11p
@$ORACLE_HOME/rdbms/admin/dbfs_create_filesystem_advanced.sql
USERS
TESTDBFS2
compress-high
deduplicate
noencrypt
partition

Dropping a File System

To drop a file system the script “dbfs_drop_filesystem.sql” must be used. Calling syntax is:

sqlplus <username>/<password>@<database>
@$ORACLE_HOME/rdbms/admin/dbfs_drop_filesystem.sql <file system name>

Usage: dbfs_client <db_user>@<db_server> [options] <mount point>

    db_user: Name of Database user that owns DBFS content store filesystem(s)
    db_server:   A valid connect string Oracle database server
                 (for example, hrdb_host:1521/hrservice).
    mount point: Path to mount Database File System(s).
                 All the file systems owned by the database user will be seen
                 at the mount point.

DBFS options:
  -o direct_io   Bypasses the Linux page cache.  Gives much better performance
                 for large files.
                 Programs in the file system cannot be executed with this option.
                 This option is recommended when DBFS is used as an ETL staging
                 area.
  -o wallet      Run dbfs_client in background. Wallet must be configured to get
                 credentials.
  -o failover    DBFS Client fails over to surviving database instance with no
                 data loss.
                 Some performance cost on writes, especially for small files.
  -o allow_root  Allows root access to the filesystem.
                 This option requires setting 'user_allow_other' parameter in
                 /etc/fuse.conf
  -o allow_other    Allows other users access to the filesystem.
                    This option requires setting 'user_allow_other' parameter in
                    /etc/fuse.conf
  -o rw             Mount the filesystem read-write [Default]
  -o ro             Mount the filesystem read-only. Files cannot be modified.
  -o trace_level=N  Trace Level: 1->DEBUG, 2->INFO, 3->WARNING, 4->ERROR,
                    5->CRITICAL [Default: 4]
  -o trace_file     <file> | 'syslog'
  -h help
  -V version

fusermount -u <mount point>

DBFS restrictions

DBFS imposes the following restrictions:

• no ioctl
• no locking
• no memory-mapped files
• no Async IOs
• no O_DIRECT file open flags
• no hard links

DBFS Client Failover

According to the oracle documentation failover can be achieved by modifying the service as shown below:

exec DBMS_SERVICE.MODIFY_SERVICE(service_name => ‘service_name’,
aq_ha_notifications => true,
failover_method => ‘BASIC’,
failover_type => ‘SELECT’,
failover_retries => 180,
failover_delay => 1);

Instead of modifying the default service a new service named “DBFS” was created and configured accordingly to the example given above.

For POLICY MANAGED Clusters issue:

srvctl add service -d ORA11P -s DBFS -g Default -y AUTOMATIC
-e SELECT -m BASIC -w 1 -z 180 -q TRUE

(“-g Default” specifies the name of the cluster group)

For ADMIN MANAGED clusters issue:

srvctl add service -d ORA11P -s DBFS -r ora11p1,ora11p2 -y AUTOMATIC
-e SELECT -m BASIC -w 1 -z 180 -q TRUE

(“-r ora11p1,ora11p2″ specified the instances the service runs)

Mounting the file system requires the parameter “-o failover” to be added:

ora11p@rac1#>: dbfs_client <db_user>@<db_server> -o failover /mnt/dbfs

-bash-3.2$ id
uid=500(grid) gid=500(dba) groups=500(dba)
-bash-3.2$ df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/sda3              26G   15G  9.6G  60% /
/dev/sda1              99M   12M   83M  13% /boot
tmpfs                 3.0G  1.4G  1.7G  45% /dev/shm
/dev/asm/ora11p_home-132 10G  4.7G  5.4G  47%
                   /u01/app/oracle/product/11.2.0/ora11p

-bash-3.2$ id
uid=501(ora11p) gid=500(dba) groups=500(dba)
-bash-3.2$ df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/sda3              26G   15G  9.6G  60% /
/dev/sda1              99M   12M   83M  13% /boot
tmpfs                 3.0G  1.4G  1.7G  45% /dev/shm
/dev/asm/ora11p_home-132 10G  4.7G  5.4G  47%
                   /u01/app/oracle/product/11.2.0/ora11p
dbfs                   14M  3.3M   11M  25% /test4

[root@rac1 ~]# id
uid=0(root) gid=0(root) groups=0(root),1(bin),2(daemon),3(sys),4(adm),6(disk),10(wheel)
[root@rac1 ~]# df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/sda3              26G   15G  9.6G  60% /
/dev/sda1              99M   12M   83M  13% /boot
tmpfs                 3.0G  1.4G  1.7G  45% /dev/shm
/dev/asm/ora11p_home-132 10G  4.7G  5.4G  47%
                    /u01/app/oracle/product/11.2.0/ora11p

[root@rac1 ~]# id
uid=0(root) gid=0(root) groups=0(root),1(bin),2(daemon),3(sys),4(adm),6(disk),10(wheel)
[root@rac1 ~]# cd /test4
-bash: cd: /test4: Permission denied

-bash-3.2$ dbfs_client dbfs2/dbfs2@rac1:1521/ora11p /test
dbfs_client: error while loading shared libraries:
libclntsh.so.11.1: cannot open shared object file: No such file or directory

ora11p@rac1#>: ldd $ORACLE_HOME/bin/dbfs_client
libstdc++.so.6 => /usr/lib64/libstdc++.so.6 (0x000000303f400000)
libfuse.so.2 => /usr/lib/libfuse.so.2 (0x00002b5e455cb000)
libclntsh.so.11.1 => not found
libnnz11.so => not found
libdl.so.2 => /lib64/libdl.so.2 (0x000000302ca00000)
libm.so.6 => /lib64/libm.so.6 (0x000000302c600000)
libpthread.so.0 => /lib64/libpthread.so.0 (0x000000302ce00000)
libnsl.so.1 => /lib64/libnsl.so.1 (0x0000003030200000)
libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x000000303c400000)
libc.so.6 => /lib64/libc.so.6 (0x000000302c200000)
librt.so.1 => /lib64/librt.so.1 (0x000000302d600000)
/lib64/ld-linux-x86-64.so.2 (0x000000302be00000)

ora11p@host#>: export LD_LIBRARY_PATH=$ORACLE_HOME/lib
ora11p@host#>: ldd dbfs_client
libstdc++.so.6 => /usr/lib64/libstdc++.so.6 (0x000000303f400000)
libfuse.so.2 => /usr/lib/libfuse.so.2 (0x00002b84ba9d9000)
libclntsh.so.11.1 => /u01/app/oracle/product/11.2.0/ora11p/lib/libclntsh.so.11.1 (0x00002b84babf8000)
libnnz11.so => /u01/app/oracle/product/11.2.0/ora11p/lib/libnnz11.so (0x00002b84bd223000)
libdl.so.2 => /lib64/libdl.so.2 (0x000000302ca00000)
libm.so.6 => /lib64/libm.so.6 (0x000000302c600000)
libpthread.so.0 => /lib64/libpthread.so.0 (0x000000302ce00000)
libnsl.so.1 => /lib64/libnsl.so.1 (0x0000003030200000)
libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x000000303c400000)
libc.so.6 => /lib64/libc.so.6 (0x000000302c200000)
librt.so.1 => /lib64/librt.so.1 (0x000000302d600000)
/lib64/ld-linux-x86-64.so.2 (0x000000302be00000)
libaio.so.1 => /usr/lib64/libaio.so.1 (0x00002b84bd5ed000)

-bash-3.2$ dbfs_client dbfs2@rac1:1521/ora11p /test
Password:
<WAIT>

echo "dbfs2" > dbfs2password

ora11p@host#>: nohup dbfs_client dbfs2@rac1:1521/ora11p /test  < dbfs2password &
[1] 24237
ora11p@host#>: nohup: appending output to `nohup.out'

ora11p@host#>: df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/sda3              26G   15G  9.6G  60% /
/dev/sda1              99M   12M   83M  13% /boot
tmpfs                 3.0G  1.4G  1.7G  45% /dev/shm
/dev/asm/ora11p_home-132  10G  4.7G  5.4G  47%
                     /u01/app/oracle/product/11.2.0/ora11p
dbfs                  6.4M  192K  6.2M   3% /test

Using the file system

Readers might ask: “Does the file system behave like a normal file system like ext3 do?” – the answer is: YES.

A few screenshots to illustrate it:

Testing de-duplication and compression

When creating the file system we specified high compression and deduplication. Lets see the results:

Compression

For this test we took the file “products.xml” from the 11g Release 2 database install source. The file is approx 930 KB in size but due to xml format good compressible.

ora11p@host#>: df -h .
Filesystem            Size  Used Avail Use% Mounted on
dbfs                  7.4M  272K  7.1M   4% /test

ora11p@host#>: cp /raw_software/database/stage/products.xml  .

ora11p@host#>: ll
total 910
-rwxr-xr-x 1 ora11p dba 931475 Oct  8 14:19 products.xml

ora11p@host#>: df -h .
Filesystem            Size  Used Avail Use% Mounted on
dbfs                  7.4M  352K  7.0M   5% /test

Although the file size is 930 KB the used space has increased from 272 KB to 352 KB – an increase by 80 KB. So the file size has been reduced from 930 KB down to 80 KB.

Deduplication

For testing de-duplication we copied the same file (products.xml) we copied in the previous example a dozen time and check the allocated space:

ora11p@host#>: ll
total 15470
-rwxr-xr-x 1 ora11p dba 931475 Oct  8 14:26 1
-rwxr-xr-x 1 ora11p dba 931475 Oct  8 14:27 10
-rwxr-xr-x 1 ora11p dba 931475 Oct  8 14:27 11
-rwxr-xr-x 1 ora11p dba 931475 Oct  8 14:27 12
-rwxr-xr-x 1 ora11p dba 931475 Oct  8 14:27 13
-rwxr-xr-x 1 ora11p dba 931475 Oct  8 14:27 14
-rwxr-xr-x 1 ora11p dba 931475 Oct  8 14:27 15
-rwxr-xr-x 1 ora11p dba 931475 Oct  8 14:27 16
-rwxr-xr-x 1 ora11p dba 931475 Oct  8 14:26 2
-rwxr-xr-x 1 ora11p dba 931475 Oct  8 14:26 3
-rwxr-xr-x 1 ora11p dba 931475 Oct  8 14:26 4
-rwxr-xr-x 1 ora11p dba 931475 Oct  8 14:26 5
-rwxr-xr-x 1 ora11p dba 931475 Oct  8 14:26 6
-rwxr-xr-x 1 ora11p dba 931475 Oct  8 14:26 7
-rwxr-xr-x 1 ora11p dba 931475 Oct  8 14:27 8
-rwxr-xr-x 1 ora11p dba 931475 Oct  8 14:27 9
-rwxr-xr-x 1 ora11p dba 931475 Oct  8 14:19 products.xml
ora11p@host#>: df -h .
Filesystem            Size  Used Avail Use% Mounted on
dbfs                   12M  704K   11M   7% /test

Based on the directory listing above we copied the products.xml 16 times. When compressing the files only each file should add approx 80 KB to the file system. Sixteen files would add up to approx 1.3 MB. But the file systems current size is only 704 KB. So there must be sime sort of de-duplication (or en extremely intelligent compression algorithm) -  but remember: this is only a first and short test. I will investigate this further in a next post.

Implementing Failover

Before testing failover we need to create an extra service which will be configured according to the guidelines from oracle:

srvctl add service -d ORA11P -s DBFS -r ora11p1,ora11p2
-y AUTOMATIC -e SELECT -m BASIC -w 1 -z 180 -q TRUE

srvctl start service -d ORA11P -s DBFS

To use this service add a connection descriptor to TNSNAMES.ORA:

DBFS =
 (DESCRIPTION =
 (ADDRESS = (PROTOCOL = TCP)(HOST = rac-scan)(PORT = 1521))
 (CONNECT_DATA =
 (SERVER = DEDICATED)
 (SERVICE_NAME = DBFS)
 )
 )

Now mount the file system normally. In our case we mount the file system on host “rac2″ while connected to host “rac1″ (inst_id=1)

-bash-3.2$ date
 Thu Oct  8 16:03:54 CEST 2009
 -bash-3.2$ ll
 total 2730
 -rwxr-xr-x 1 ora11p dba 931475 Oct  8 14:26 1
 -rwxr-xr-x 1 ora11p dba 931475 Oct  8 14:27 9
 -rwxr-xr-x 1 ora11p dba 931475 Oct  8 14:19 products.xml

SQL> select inst_id, username, to_char(logon_time,'dd.mm.yyyy hh24:mi:ss') as
 TIME from gv$session where username='DBFS2'

INST_ID USERNAME  TIME
 ---------- --------- -------------------
 1 DBFS2     08.10.2009 15:54:17

We now do a “shutdown abort” on instance “ora11p1″ (i.e. the instance running on host “rac1″ and the instance the dbfs is connected to) and check the file system and sessions:

Shutdown the instance “ora11p1″:

SQL@ORA11P!> shutdown abort;

Alert.log information from instance “ora11p2″ showing instance “ora11p1″ was shut down at 16:05:26:

Thu Oct 08 16:05:26 2009
Reconfiguration started (old inc 28, new inc 30)
List of instances:
 2 (myinst: 2)
 Global Resource Directory frozen
 * dead instance detected - domain 0 invalid = TRUE
 Communication channels reestablished
 Master broadcasted resource hash value bitmaps
 Non-local Process blocks cleaned out
Thu Oct 08 16:05:26 2009
 LMS 0: 0 GCS shadows cancelled, 0 closed, 0 Xw survived
 Set master node info
 Submitted all remote-enqueue requests
 Dwn-cvts replayed, VALBLKs dubious
 All grantable enqueues granted
 Post SMON to start 1st pass IR
Thu Oct 08 16:05:26 2009
Instance recovery: looking for dead threads
Beginning instance recovery of 1 threads
 Submitted all GCS remote-cache requests
 Post SMON to start 1st pass IR
 Fix write in gcs resources
Reconfiguration complete
Started redo scan
Completed redo scan
 read 1725 KB redo, 701 data blocks need recovery
Started redo application at
 Thread 1: logseq 454, block 1674
Recovery of Online Redo Log: Thread 1 Group 2 Seq 454 Reading mem 0
 Mem# 0: +DATA/ora11p/onlinelog/group_2.258.698489181
 Mem# 1: +DATA2/ora11p/onlinelog/group_2.260.698489185
Completed redo application of 1.01MB
Completed instance recovery at
 Thread 1: logseq 454, block 5124, scn 5701524
 561 data blocks read, 723 data blocks written, 1725 redo k-bytes read
Thread 1 advanced to log sequence 455 (thread recovery)

Session Information on instance “ora11p2″:

SQL> select inst_id, username, to_char(logon_time,'dd.mm.yyyy hh24:mi:ss')
from gv$session where username='DBFS2'

 INST_ID   USERNAME   TO_CHAR(LOGON_TIME,
---------- ---------- -------------------
 2         DBFS2      08.10.2009 16:05:29

The session was reconnected to the surviving instance three seconds after the instance “ora11p1″ was shut down.

Listing the file system was interrupted shortly as shown below:

-bash-3.2$ date
Thu Oct  8 16:05:28 CEST 2009
-bash-3.2$ ll

<< WAITED a few seconds >>

total 2730
-rwxr-xr-x 1 ora11p dba 931475 Oct  8 14:26 1
-rwxr-xr-x 1 ora11p dba 931475 Oct  8 14:27 9
-rwxr-xr-x 1 ora11p dba 931475 Oct  8 14:19 products.xml
-bash-3.2$ date
Thu Oct  8 16:05:36 CEST 2009

However if you have not configured SSH equivalence before this short guide shows how to do set up ssh equivalence between the nodes “rac1.regner.de” and “rac2.regner.de”:

1. Unpack the database installation files

2. Execute:

<PATH_TO_DATABASE_INSTALLATION>/sshsetup/sshUserSetup.sh
  -user ora11p -hosts "rac1.regner.de rac2.regner.de" -shared

Arguments:
-user: the name of the user for ssh equivalence to be configured
-hosts: space separated list of node names
-shared: indicates a shared oracle home (on acfs or nfs)

3. Check it

/usr/bin/ssh -o FallBackToRsh=no  -o PasswordAuthentication=no 
  -o StrictHostKeyChecking=yes  -o NumberOfPasswordPrompts=0 rac2 /bin/date

/usr/bin/ssh -o FallBackToRsh=no  -o PasswordAuthentication=no
  -o StrictHostKeyChecking=yes  -o NumberOfPasswordPrompts=0 rac1 /bin/date

The two commands above should return the current date. Thats it.

• Oracle database and grid infrastructure binaries installed
• at least ONE or better two disk groups configured for:
  • database and binary files
  • flash recovery area

Our database name will be “ORA11P”.

Preparations

• create directory for cfgtools

create /u01/app/oracle/cfgtoollogs
mkdir -p /u01/app/oracle/cfgtoollogs
chown root:dba /u01/app/oracle/cfgtoollogs
chmod 775 /u01/app/oracle/cfgtoollogs

• Check SSH equivalence once again:
  execute as user „ora11p“ (this is the user holding the oracle database binary installation):

/usr/bin/ssh -o FallBackToRsh=no  -o PasswordAuthentication=no 
  -o StrictHostKeyChecking=yes  -o NumberOfPasswordPrompts=0 rac2 /bin/date

/usr/bin/ssh -o FallBackToRsh=no  -o PasswordAuthentication=no 
  -o StrictHostKeyChecking=yes  -o NumberOfPasswordPrompts=0 rac1 /bin/date

Both commands should return the current date. If there are errors like „connection refused“, check:

• network connectivity between both nodes
• does a „normal“ SSH connection works?:
  • ssh root@rac1-priv
  • ssh root@rac1
  • ssh root@rac2-priv
  • ssh root@rac2

both commands should prompt for a password.

If SSH was working before and seems “gone” you might face a problem documented here.

• add ACFS oracle home to cluster registry

$GRID_HOME/bin/srvctl add filesystem -d /dev/asm/ora11p_home-132
  -v ora11p_home -g DATA2 -m /u01/app/oracle/product/11.2.0/ora11p -u ora11p

Arguments:
-d: path of advm volume
-v: name of volume
-m: mount point path

„-u ora11p“ ist important and must match the owner of the database binary installation; else dbca will raise error compaining about missing permissions

Create the database with dbca

Start dbca

Where to go now?

• Tune Instance, for instance:
  • memory_target
  • db_writer_processes
  • sessions parameter
  • …
• Configure, perform and test your backup

SCAN on the server side

When installing a 11g Release 2 grid infrastructure you are asked for the cluster name which will be part of the SCAN. The notation for the SCAN is:

<clustername>-scan.<domain>

For instance if your cluster is named “rac” and the domain “regner.de” the SCAN name will be “rac-scan.regner.de”.

In order to successful install grid infrastructure you need to configure your DNS (hosts file entries will not work!) prior installing grid infrastructure to resolve the name accordingly. Oracle requires at least one, better three IPs configured for the scan name. Your DNS zone file might look like this:

In the example zone file above we configured three IPs for the scan: 172.23.15.3, 172.23.15.4 and 172.23.15.5.

After installation you will find three listener processes running (separated on all cluster nodes) as shown in the following figure:

Listener for SCAN2 and SCAN3 are running on node “rac1″ and listener for SCAN1 is running on node “rac2″. For each SCAN listener there will be a dedicated network interface running with the same IP as configured in DNS:

Client connections

Connection to database “RAC11P” using SCAN would use this tnsnames entry:

RAC11P =
 (DESCRIPTION=
 (ADDRESS=(PROTOCOL=tcp)(HOST=rac-scan.regner.de)(PORT=1521))
 (CONNECT_DATA=(SERVICE_NAME=RAC11P))
 )

The ”old fashioned” way still works:

RAC11P_old =
 (DESCRIPTION=
 (ADDRESS_LIST=
 (ADDRESS=(PROTOCOL=tcp)(HOST=rac1-vip.regner.de)(PORT=1521))
 (ADDRESS=(PROTOCOL=tcp)(HOST=rac2-vip.regner.de)(PORT=1521))
 )
 (CONNECT_DATA=(SERVICE_NAME=RAC11P))
 )

Connecting to a named instance:

RAC11P =
 (DESCRIPTION=
 (ADDRESS=(PROTOCOL=tcp)(HOST=rac-scan.regner.de)(PORT=1521))
 (CONNECT_DATA=(SERVICE_NAME=RAC11P)
 (INSTANCE_NAME=RAC11P1))
 )