While testing i came to the conclusion that with the help of clones oracle databases (and, of course – other databases as well) can be clones extremely fast with almost no disk space required. So i tested it.

Read on for the results…

Setup everything

Prior testing snapshots i had to install an oracle database and had to create a database. This wont be covered here.

The ZFS file system structure looks like this:

root@oracle:~# df -h
Filesystem             size   used  avail capacity  Mounted on
/                        19T    20G    19T     1%    /
/dev                      0K     0K     0K     0%    /dev
proc                      0K     0K     0K     0%    /proc
ctfs                      0K     0K     0K     0%    /system/contract
mnttab                    0K     0K     0K     0%    /etc/mnttab
objfs                     0K     0K     0K     0%    /system/object
swap                     49G   176K    49G     1%    /etc/svc/volatile
/usr/lib/libc/libc_hwcap1.so.119T    20G    19T     1%   
/lib/libc.so.1fd          0K     0K     0K     0%    /dev/fd
swap                     49G     4K    49G     1%    /tmp
swap                     49G    12K    49G     1%    /var/run
pool1/zones/oracle/db01  27T    52K    19T     1%    /u01/oradata/ORA10P

From the output above you will most probably recognize the path for storing our SOURCE database: /u01/oradata/ORA10P.

In this article i will clone the database with the SID “ORA10P” located in “/u01/oradata/ORA10P” to a database with SID “ORA11P” located in “/u01/oradata/ORA11P”.

Cloning the database

Cloning the database uses the ability of ZFS to create snapshots and to mount them as a writable file system. In addition to that we clone an oracle database by re-creating the control files and changing the database SID.

In the following i will outline the required steps:

Step 1 – Create the Snapshot

For the first tests i created the snapshot while the database was closed:

root@oracle:~# zfs snapshot pool1/zones/oracle/db01@db_clone

Step 2 – Create and Mount the Clone

Create a writeable snapshot called “clone”:

zfs clone pool1/zones/oracle/db01@db_clone pool1/zones/oracle/db02

Set Mountpoint for that clone:

zfs set mountpoint=/u01/oradata/ORA11P pool1/zones/oracle/db02

Thats it. With these steps you created a writable snapshop of your database and mounted it at /u01/oradata/ORA11P.

Lets take a look at the used space:

root@oracle:~# zfs list
NAME                          USED  AVAIL  REFER  MOUNTPOINT
pool1                        7.92T  18.7T  59.6K  /pool1
pool1/zones/oracle/db01      1011M  18.7T   912M  /u01/oradata/ORA10P
pool1/zones/oracle/db02          0  18.7T   912M  /u01/oradata/ORA11P

According to ZFS data stored in /u02/oradata/ORA11P does not take up any space at all… so it is empty?

No it´s not:

root@oracle:~# ls -la  /u01/oradata/ORA11P
total 1867170
drwxrwxr-x   2 ora10p   dba           13 Feb 17 15:00 .
drwxrwxr-x   4 ora10p   dba            4 Feb 17 15:09 ..
-rw-r-----   1 ora10p   dba      7061504 Feb 17 15:02 control01.ctl
-rw-r-----   1 ora10p   dba      7061504 Feb 17 15:02 control02.ctl
-rw-r-----   1 ora10p   dba      7061504 Feb 17 15:02 control03.ctl
-rw-r-----   1 ora10p   dba      52429312 Feb 17 15:01 redo01.log
-rw-r-----   1 ora10p   dba      52429312 Feb 17 15:01 redo02.log
-rw-r-----   1 ora10p   dba      52429312 Feb 17 15:02 redo03.log
-rw-r-----   1 ora10p   dba      251666432 Feb 17 15:02 sysaux01.dbf
-rw-r-----   1 ora10p   dba      492838912 Feb 17 15:02 system01.dbf
-rw-r-----   1 ora10p   dba      20979712 Feb 17 15:00 temp01.dbf
-rw-r-----   1 ora10p   dba      26222592 Feb 17 15:02 undotbs01.dbf
-rw-r-----   1 ora10p   dba      5251072 Feb 17 15:02 users01.dbf

The space shown by “zfs list” shows the occupied amount of space. Referenced blocks are not counted. Immediate after creating the clone all blocks in the clone refer to the original blocks thus taking up no space at all.

Step 3 – Backup the source database control file to trace

As SYSDBA connected to your SOURCE database (in our case “ORA10P” is our source database) execute:

SQL>
SQL> alter database backup controlfile to trace;

Database altered.

After that locate the trace file just created. Look for the trace file in the directory you specified as database parameter “user_dump_dest”.

At my site the parameter is set to “/u01/app/oracle/admin/ORA10P/udump”. So i have to look for the trace file there.

-bash-4.0$ ls -lat  /u01/app/oracle/admin/ORA10P/udump
total 90
-rw-r—–   1 ora10p   dba         6044 Feb 17 15:16 ora10p_ora_21829.trc
drwxr-x—   2 ora10p   dba           13 Feb 17 15:16 .
-rw-r—–   1 ora10p   dba          625 Feb 17 15:16 ora10p_ora_21824.trc
-rw-r—–   1 ora10p   dba          569 Feb 17 15:16 ora10p_ora_21795.trc
-rw-r—–   1 ora10p   dba          755 Feb 17 15:02 ora10p_ora_20396.trc

Most probably the file i am looking for is “ora10p_ora_21829.trc” because i looked for the file immediate after backing up the control file to trace.

Step 4 – Edit the control file

I have successfully located the control file trace dump and copied the file:

-bash-4.0$ cp  ora10p_ora_21829.trc create.sql

The file header looks like this:

/u01/app/oracle/admin/ORA10P/udump/ora10p_ora_21829.trc
Oracle Database 10g Enterprise Edition Release 10.2.0.1.0 - Production
With the Partitioning, OLAP and Data Mining options
ORACLE_HOME = /u01/app/oracle/product/ora10p
System name:    SunOS
Node name:      oracle
Release:        5.11
Version:        snv_132
Machine:        i86pc
Instance name: ORA10P
Redo thread mounted by this instance: 1
Oracle process number: 23
Unix process pid: 21829, image: oracle@oracle (TNS V1-V3)

*** SERVICE NAME:(SYS$USERS) 2010-02-17 15:16:57.065
*** SESSION ID:(159.1) 2010-02-17 15:16:57.065
*** 2010-02-17 15:16:57.065
-- The following are current System-scope REDO Log Archival related
-- parameters and can be included in the database initialization file.
--
-- LOG_ARCHIVE_DEST=''
-- LOG_ARCHIVE_DUPLEX_DEST=''

[....]

[....]

--     Set #1. NORESETLOGS case
--
-- The following commands will create a new control file and use it
-- to open the database.

[....]

For editing execute the following steps:

1. Locate the section starting with”–     Set #2. RESETLOGS case”
2. Skip forward until you find:
   CREATE CONTROLFILE REUSE DATABASE “ORA10P” RESETLOGS  NOARCHIVELOG

   Note that the database name might be different!

3. DELETE EVERYTHING ABOVE THE LINE STARTING WITH “CREATE DATABASE”
4. Edit that line as follows:
   CREATE CONTROLFILE SET DATABASE “ORA11P” RESETLOGS  NOARCHIVELOG
5. Skip to the file paths:
   [...]
   LOGFILE
   GROUP 1 ‘/u01/oradata/ORA10P/redo01.log’  SIZE 50M,
   GROUP 2 ‘/u01/oradata/ORA10P/redo02.log’  SIZE 50M,
   GROUP 3 ‘/u01/oradata/ORA10P/redo03.log’  SIZE 50M
   – STANDBY LOGFILE
   DATAFILE
   ‘/u01/oradata/ORA10P/system01.dbf’,
   ‘/u01/oradata/ORA10P/undotbs01.dbf’,
   ‘/u01/oradata/ORA10P/sysaux01.dbf’,
   ‘/u01/oradata/ORA10P/users01.dbf’
   CHARACTER SET WE8ISO8859P1
   ;
6. Edit them to reflect the new paths:
   [...]
   LOGFILE
   GROUP 1 ‘/u01/oradata/ORA11P/redo01.log’  SIZE 50M,
   GROUP 2 ‘/u01/oradata/ORA11P/redo02.log’  SIZE 50M,
   GROUP 3 ‘/u01/oradata/ORA11P/redo03.log’  SIZE 50M
   – STANDBY LOGFILE
   DATAFILE
   ‘/u01/oradata/ORA11P/system01.dbf’,
   ‘/u01/oradata/ORA11P/undotbs01.dbf’,
   ‘/u01/oradata/ORA11P/sysaux01.dbf’,
   ‘/u01/oradata/ORA11P/users01.dbf’
   CHARACTER SET WE8ISO8859P1
   ;
   Note: This is the most important part here. You have to make sure you do not reference any file used by the source database here!
7. Delete EVERYTHING BELOW

Step 5 – Clone the database by re-creating the control file

After editing the script it should look like the following sample for re-naming the source database with SID “ORA10P” to “ORA11P”:

CREATE CONTROLFILE SET DATABASE "ORA11P" RESETLOGS  NOARCHIVELOG
 MAXLOGFILES 16
 MAXLOGMEMBERS 3
 MAXDATAFILES 100
 MAXINSTANCES 8
 MAXLOGHISTORY 292
LOGFILE
 GROUP 1 '/u01/oradata/ORA11P/redo01.log'  SIZE 50M,
 GROUP 2 '/u01/oradata/ORA11P/redo02.log'  SIZE 50M,
 GROUP 3 '/u01/oradata/ORA11P/redo03.log'  SIZE 50M
-- STANDBY LOGFILE
DATAFILE
 '/u01/oradata/ORA11P/system01.dbf',
 '/u01/oradata/ORA11P/undotbs01.dbf',
 '/u01/oradata/ORA11P/sysaux01.dbf',
 '/u01/oradata/ORA11P/users01.dbf'
CHARACTER SET WE8ISO8859P1
;

If neccessary create a new pfile for the new instance and the directories specified in “audit_file_dest”, “background_dump_dest” and “user_dump_dest”. Just make sure you change:

• control file location
• db_name

A simple pfile for our new database looks like this:

audit_file_dest='/u01/app/oracle/admin/ORA11P/adump'
background_dump_dest='/u01/app/oracle/admin/ORA11P/bdump'
compatible='10.2.0.1.0'
control_files='/u01/oradata/ORA11P/control01.ctl','/u01/oradata/ORA11P/control02.ctl','/u01/oradata/ORA11P/control03.ctl'
core_dump_dest='/u01/app/oracle/admin/ORA11P/cdump'
db_block_size=8192
db_file_multiblock_read_count=16
db_name='ORA11P'
open_cursors=300
pga_aggregate_target=134217728
processes=150
remote_login_passwordfile='EXCLUSIVE'
sga_target=536870912
undo_management='AUTO'
undo_tablespace='UNDOTBS1'
user_dump_dest='/u01/app/oracle/admin/ORA11P/udump'

After setting everything everything up DELETE the control files of the source database located in the clone file system “/u01/oradata/ORA11P“. DO NOT DELETE THE CONTROL FILES OF YOUR SOURCE DATABASE IN “/u01/oradata/ORA10P”!!

Afterwards tart the instance:

-bash-4.0$ export ORACLE_SID=ORA11P
-bash-4.0$ sqlplus "/ as sysdba"
SQL*Plus: Release 10.2.0.1.0 - Production on Wed Feb 17 15:50:34 2010
Copyright (c) 1982, 2005, Oracle.  All rights reserved.
Connected to an idle instance.
SQL> startup nomount
ORACLE instance started.
Total System Global Area  536870912 bytes
Fixed Size                  2120416 bytes
Variable Size             152937760 bytes
Database Buffers          377487360 bytes
Redo Buffers                4325376 bytes
SQL>

If the instance starts successfully with “startup nomount” execute our just created script:

SQL> @create
Control file created.

Note: If you would have not deleted the control files the script will raise an error it cannot re-create the control files because they are already there:

SQL> @create
CREATE CONTROLFILE SET DATABASE "ORA11P" RESETLOGS  NOARCHIVELOG
*
ERROR at line 1:
ORA-01503: CREATE CONTROLFILE failed
ORA-00200: control file could not be created
ORA-00202: control file: '/u01/oradata/ORA11P/control01.ctl'
ORA-27038: created file already exists
Additional information: 1

The reason for the control files being already there is that we cloned everything including redo logs, temporary tablespace and also control files.

After re-creating the contron file issue:

SQL> alter database open resetlogs;
Database altered.
SQL> update global_name set global_name='ORA11P';
1 row updated.
SQL> commit;
Commit complete.
SQL> alter tablespace temp add tempfile '/u01/oradata/ORA11P/temp01.dbf' reuse;
Tablespace altered.

After that shut down the instance and startup again to check if everything is OK:

SQL> shutdown immediate
Database closed.
Database dismounted.
ORACLE instance shut down.

SQL> startup
ORACLE instance started.
Total System Global Area  536870912 bytes
Fixed Size                  2120416 bytes
Variable Size             152937760 bytes
Database Buffers          377487360 bytes
Redo Buffers                4325376 bytes
Database mounted.
Database opened.

After finishing these steps you have a bit wise identical copy of your source database. In addition to that i strongly recommend to check the alert.log for any errors.

Checking Space again

After cloning our database we observe an increase in storage needs for our cloned database:

pool1/zones/oracle/db01      1.02G  18.7T   912M  /u01/oradata/ORA10P
pool1/zones/oracle/db02      81.2M  18.7T   913M  /u01/oradata/ORA11P

After performing the steps above the just re-created control files and redo logs take 81.2 MB space because these blocks are not shared anymore.

The great picture

The example above is just a proof-of-concept. It shows there can be significant storage savings by using ZFs snapshot/cloning mechanism for instance to create database or development or testing purposes.

In addition to that cloing the database is extremely fast and does not depend on the size of the database being cloned. The whole procedure outlined here takes when done manually approx. 5 minutes regardless of the database size.

Most probably (i have not tested this yet) the database can be cloned online by putting the database in hot backup mode (“alter database backup begin”) before taking the snapshot.

And – most important – it is COMPLETELY SCRIPTABLE!

Part II left with a current status of a crashing machine during high I/O loads.During the previous weeks i tried to solve the stability problems. And finally: we solved them!

This third part is about our problems, what caused them and how we finally solved them….

Problems, Problems

Part I – using Adaptec Controller

Initially we equipped our storage with one Adaptec 52445 or one Adaptec 51645. These adapters were chosen because they offered a large amount of SATA ports (16 and 24) so we could attach all remaining disks directly without any expander.

During our tests we noticed a compete halt of all I/O under high I/O load. Not even a “ls” returns any result. The problem was discussed at the opensolaris mailing list but without any insight. Even the message log did not showed any error message.

I tried several settings on controller side, including creating a RAID array, turning off NCQ, setting timeout values higher and so on. But nothing worked. Under load sooner or later the I/O to all drives attached to the adaptec controller came to a complete halt.

After noticing that high I/O to disks directly attached to the motherboard do not “lock up” in any way we were pretty sure out problems were caused by the adaptec controller…. so we replaced them.

Part II – using LSI 1086E-based controllers

In order to solve our problems which were most probably related to the adaptec controllers we replaced them with two LSI 1086E-based controllers plus one expander attached to each controller. We used the Chenbro CK12804 expander.

Unfortunately we observed the same behavior with these controllers as well: the I/O stalled sooner or later.

So i took a look at the hard disks next. For the prototype we ordered fourty Seagate ST31000340NS disks with 1 TB each. Note that these disks are – according to the seagate homepage – designed for:

"The Barracuda ES.2 drive is the perfect solution for high-capacity enterprise storage
applications such as the migration of mission-critical transactional data, from tier 1
to tier 2 (nearline) storage, where dollars/GB and GB/watt are a primary concern."

Another criteria for chosing this drive was:

"- 24x7 operation and 1.2 M hrs. MTBF"

After making sure we used the most recent firmware (actually “SN06″ is the most recent firmware) i took a look at the Seagate forums and found an interesting thread about Seagate ES.2 drives (the drive we used) and some firmware problems. Especially one post took my attention:

I've got a setup with two 1TB disks which previously used the AN05 firmware and they
worked just fine, no matter how many GB's of data I copied to/from the RAID1 mirror.
Right after updating the firmware to suggested SN06 version it seems that I must
disable NCQ completely or the system will hang in 30 mins if there are any data being
copied from the disks. Because there has not been any guarantees that AN05-version of
the firmware doesn't suffer from the disk locking bug, I did not feel comfortable in
continuing using that version even though it seemed to work just fine.

Well, i thought to myself: “You´ve already tried disabling NCQ at /etc/system level and controller level with the adaptec controller but anyway – try it!”. So i turned off NCQ at LSI controllers with the help of LSIutil.

After booting the system again i put some I/O load on it and waited for the problem to appear again…. after waiting pessimistically for days it did not appear. So i once again booted the system and increased the I/O pressure and waited again. Soon after some errors were shown:

Jan 29 12:00:48 openstorage scsi: [ID 107833 kern.notice]
scsi_state = 0, transfer count = 1400, scsi_status = 0
Jan 29 12:00:48 openstorage scsi: [ID 107833 kern.warning] WARNING:
/pci@0,0/pci8086,340c@5/pci1014,396@0/sd@a,0 (sd52):
Jan 29 12:00:48 openstorage     incomplete read- retrying
Jan 29 12:00:48 openstorage scsi: [ID 107833 kern.warning] WARNING:
/pci@0,0/pci8086,340c@5/pci1014,396@0 (mpt1):
Jan 29 12:00:48 openstorage     unknown ioc_status = 4
Jan 29 12:00:48 openstorage scsi: [ID 107833 kern.notice]
scsi_state = 0, transfer count = 12800, scsi_status = 0
Jan 29 12:00:48 openstorage scsi: [ID 107833 kern.warning] WARNING:
/pci@0,0/pci8086,340c@5/pci1014,396@0/sd@8,0 (sd50):
Jan 29 12:00:48 openstorage     incomplete read- retrying

Despite of these errors a complete lockup of all I/O was not observed anymore. So disabling NCQ did the trick as the following days (and weeks) proved.

Obviously Seagates SATA “server” disks have some kind of problem with NCQ. Using an older firmware was not an option because older firmware versions might cause the disks to fail completely. A newer firmware than SN06 was also not available. I wont comment this any further but most likely i will think twice before buying any Seagate disks again. I even contacted Seagate support but i have not heared anything from them.

Interestingly disabling NCQ at the Adaptec controller and even on /etc/system level did not work. I have not investigated this further.

Part III – getting rid of “incomplete read- retrying” messages

After replacing the adaptec controller with some LSI controller and disabling NCQ for all disks the system stabilized. Even extremely high I/O would not lock up anymore.

But there were still error messages like this:

Jan 29 12:00:48 openstorage     incomplete read- retrying
Jan 29 12:00:48 openstorage scsi: [ID 107833 kern.warning] WARNING:
/pci@0,0/pci8086,340c@5/pci1014,396@0 (mpt1):
Jan 29 12:00:48 openstorage     unknown ioc_status = 4

When debugging these errors i noticed every time an “incomplete read- retrying” message was issued the error counters on the physical link level was increased:

Adapter Phy 6:  Link Up
 Invalid DWord Count                                      1
 Running Disparity Error Count                            1
 Loss of DWord Synch Count                                0
 Phy Reset Problem Count                                  0

<INCOMPLETE READ ERROR MESSAGE>

Adapter Phy 6:  Link Up
 Invalid DWord Count                                      2
 Running Disparity Error Count                            2
 Loss of DWord Synch Count                                0
 Phy Reset Problem Count                                  0

So it seemed related either to the physical connection – or – still the hard disks. Before digging further on hard disk level we replaced the cables with new ones. The error persisted.

In the thread quoted above the user not only reported problems with NCQ but changed the SATA interface speed from 3 gbit/s to 1.5 gbit/s as well. According to his findings changing the speed is not a viable workaround. But i tried it anyway. After chaning the speed of the drives interface to 1.5 Gbit/s the error disappeared as well.

The controller configuration looks like this:

Main menu, select an option:  [1-99 or e/p/w or 0 to quit] 16

SAS1068E's links are 1.5 G, 1.5 G, 1.5 G, 1.5 G, 3.0 G, 3.0 G, 3.0 G, down
 B___T     SASAddress     PhyNum  Handle  Parent  Type
        500605b0017ff110           0001           SAS Initiator
        500605b0017ff111           0002           SAS Initiator
        500605b0017ff112           0003           SAS Initiator
        500605b0017ff113           0004           SAS Initiator
        500605b0017ff114           0005           SAS Initiator
        500605b0017ff115           0006           SAS Initiator
        500605b0017ff116           0007           SAS Initiator
        500605b0017ff117           0008           SAS Initiator
 0   5  09221b095e5c7c67     0     0009    0001   SATA Target
 0   6  09221b095d6f556b     1     000a    0002   SATA Target
 0   7  09221b087d585275     2     000b    0003   SATA Target
 0   8  09221b087a5a7a6c     3     000c    0004   SATA Target
        5001c450000c4700     4     000d    0005   Edge Expander
 0   9  5001c450000c470c    12     000e    000d   SATA Target
 0  10  5001c450000c470d    13     000f    000d   SATA Target
 0  11  5001c450000c470e    14     0010    000d   SATA Target
 0  12  5001c450000c470f    15     0011    000d   SATA Target
 0  13  5001c450000c4710    16     0012    000d   SATA Target
 0  14  5001c450000c4711    17     0013    000d   SATA Target
 0  15  5001c450000c4712    18     0014    000d   SATA Target
 0  16  5001c450000c4713    19     0015    000d   SATA Target
 0  17  5001c450000c4714    20     0016    000d   SATA Target
 0  18  5001c450000c4715    21     0017    000d   SATA Target
 0  19  5001c450000c4716    22     0018    000d   SATA Target
 0  20  5001c450000c4717    23     0019    000d   SATA Target
 0  21  5001c450000c4718    24     001a    000d   SATA Target
 0  22  5001c450000c4719    25     001b    000d   SATA Target
 0  23  5001c450000c471a    26     001c    000d   SATA Target
 0  24  5001c450000c471b    27     001d    000d   SATA Target
 0  25  5001c450000c473d    28     001e    000d   SAS Initiator and Target

Type      NumPhys    PhyNum  Handle     PhyNum  Handle  Port  Speed
Adapter      8          0     0001  -->    0     0009     0    1.5
  1     0002  -->    0     000a     1    1.5
  2     0003  -->    0     000b     2    1.5
  3     0004  -->    0     000c     3    1.5
  4     0005  -->    0     000d     4    3.0
  5     0005  -->    1     000d     4    3.0
  6     0005  -->    2     000d     4    3.0
Expander    30          0     000d  -->    4     0005     4    3.0
  1     000d  -->    5     0005     4    3.0
  2     000d  -->    6     0005     4    3.0
 12     000d  -->    0     000e     4    1.5
 13     000d  -->    0     000f     4    1.5
 14     000d  -->    0     0010     4    1.5
 15     000d  -->    0     0011     4    1.5
 16     000d  -->    0     0012     4    1.5
 17     000d  -->    0     0013     4    1.5
 18     000d  -->    0     0014     4    1.5
 19     000d  -->    0     0015     4    1.5
 20     000d  -->    0     0016     4    1.5
 21     000d  -->    0     0017     4    1.5
 22     000d  -->    0     0018     4    1.5
 23     000d  -->    0     0019     4    1.5
 24     000d  -->    0     001a     4    1.5
 25     000d  -->    0     001b     4    1.5
 26     000d  -->    0     001c     4    1.5
 27     000d  -->    0     001d     4    1.5
 28     000d  -->    0     001e     4    3.0

As you can see the connection Controller –> Expander has a bandwidth of 3.0 Gbit/s whereas Expander –> Drive is limited to 1.5 Gbit/s.

Conclusion

After disabling NCQ on controller lever and limiting the bandwidth between expander and hard disk to 1.5 Gbit/s for each disk all problems are gone.

All? Yes, indeed. For three weeks there are NO error messages anymore and the system itself is rock-stable. During that period one hard disk physically failed and had to be replaced (remember, i wont comment on the quality of these “server” hard disks….).

Tomorrow or the day after tomorrow i wil post Part IV covering first performance benchmarks.

Last week i successfully completed all requirements to become an

Oracle Certified Trainer

Preparing for the course took all my time so there was little time left to write in my blog. Beside preparing for the trainer tests i worked a lot on my other project: building a cheap custom storage yourself.

]]> http://blog.ronnyegner-consulting.de/2010/02/16/i-am-oracle-certified-trainer-now/feed/ 5 http://blog.ronnyegner-consulting.de/2010/01/11/building-a-custom-and-cheap-storage-server-yourself-part-ii/ http://blog.ronnyegner-consulting.de/2010/01/11/building-a-custom-and-cheap-storage-server-yourself-part-ii/#comments Mon, 11 Jan 2010 15:15:58 +0000 Ronny Egner http://blog.ronnyegner-consulting.de/?p=1552 It´s been a few months since posting the idea of building a custom made storage system in my blog. During this time we convinced people to give our idea a try, did some minor changes to the box layout and ordered the parts. Finally they arrived at 27th December 2009.

So this is part II of the project called “Building a custom and cheap storage server yourself”. For people dont know what i am talking about: part I can be found here and part III here.

Building the box

We had a budget of approx 12.000 euros for building the prototype of the storage box. We decided to build the prototype with 40 disks from the start and fit an SAN-HBA in it to try COMSTAR (this enables us to export the storage via SAN to other servers). As operating system we choose Open Solaris. Two disks are dedicated to the operating system and are attached directly to the mainboard. The remaining 38 disks are attached to either one Adaptec 52445 or one Adaptec 51645 controller.

Below are some pictures of the components and the final assembled box. Click on the image for a larger version:

Installing

After putting everything in place we started to install Open Solaris for the first time. But wait – we forgot a DVD drive. But USB-attached dvd drives are widely available. After attaching a usb-dvd drive we started to install.

This time we were able to boot from dvd but installation did not start. A short glance at the mainboard manufacturer site yielded:

Fixed an issue where the system could not boot from a DVD ROM when an Adaptec Raid Card
(ASR-5805) was installed on any PCI-e Slot

Yup, we had a similar controller installed. So – instead of shorthand installing the operating system and playing around a little bit – we patched all components to the most recent version (BIOS, HBA, and so on).

After doing so installation went fine and the system booted Open Solaris 2009.06 for the first time.

Patching (again)

Open Solaris 2009.06 is quite old and we needed the latest features of ZFS and COMSTAR so we upgraded our Open Solaris. There are basically two versions available:

• A “release” version (currently as of December 2009: Build 111) of Open Solaris (Repository Link)
• A “development” version (currently as of December 2009: Build 130) of Open Solaris (Repository Link)

The upgrade process is pretty straight forward:

Upgrading to the most recent “release” version is done by entering:

pfexec pkg image-update

Upgrading to the most recent “development” build is done by:

$ pfexec pkg set-publisher -O http://pkg.opensolaris.org/dev opensolaris.org
$ pfexec pkg image-update

We first patched to the most recent “release” build and afterwards patched to the most recent “developement” build.Beside from the painfully slow fetching of the packages (*really* slow… it run several hours for few hundred MB) everything worked fine and we ended up with two bootable configurations:

• Release – Build 111
• Development – Build 130

We continued our work with the development build due to the fact we wanted deduplication and the most recent version with the most recent fixes included.

Configure

Replacing the Adaptec controller driver

In order to use the drives attached to the Adaptec controller we had to replace the driver shipped with Open Solaris with the appropriate driver from Adaptec.

Creating the ZFS pool

After successfully booting with Build 130 and replacing the adaptec driver we created our first ZFS pool “pool1″ with a total capacity of 20 TB:

zpool create pool1 raid2z c10t0d0s0 c11t0d0s0 c12t0d0s0 c13t0d0s0 c14t0d0s0 c15t0d0s0 c16t0d0s0 c17t0d0s0
c18t0d0s0 c19t0d0s0 c20t0d0s0 c21t0d0s0 c22t0d0s0 c23t0d0s0 c24t0d0s0 c25t0d0s0 c26t0d0s0 c27t0d0s0
c28t0d0s0 c29t0d0s0 c30t0d0s0 c31t0d0s0

After several seconds the zpool was created successfully. So far – so cool.

I know a raid1z pool with 20 disks i pretty uncommon and i would never ever use this configuration in production due to the extremely high probability of a double-disk-failure. But for the very first tests it seemed acceptable.

Install missing packages

For using COMSTAR (and iSCSI as well) we needed several packages which could be installed with the help of the package manager (“pkg”) easily.

Booting after COMSTAR and ZPool creation

The first boot after installing the COMSTAR packages, replacing the qlc with the qlt driver (to export our storage over SAN) and creating the large zpool caused the system to crash during boot.

It took me some time to find out package management silently uninstalled the adaptec controller driver from adaptec we installed before and replaced it with the original Open Solaris driver (which cannot use disks attached to the adaptec controller) when installing the COMSTAR packages. Booting afterwards caused the system to crash.

Testing and Crashing

Currently we are testing overall system performance. While doing so we faced problems were I/O to the adaptec controller would hang occasionally. The system is still responsible (most probably because the system disks are not attached via the adaptec controller) but I/O to the data pool is impossible. I´ve posted the problem at the opensolaris mailing list but currently with no replies.

Conclusion: currently crashing

Although we are facing some problems we will continue this project. First of all we separated the hard disks on each controller into their own zpools. If this is a hardware issue the error should be located to one zpool only. In addition to that we will replace the controller with LSI-based ones which are also used in the SUN Thumper systems.

If the system runs performance is good: We observed up to 1 GB/s (not Gbit!) or 1000 MB/s sequential read/write speed and up to 7500 I/O operations per second. Exporting storage via SAN works as well with decent speed: We observed up to 320 MB/s on a QLogic 4 Gbit/s HBA running under Linux SLES 10 SP3.

• HP-UX (Itanium) and
• AIX (PPC64)

I am surprised 11g R2 for Itanium was released prior 11g Release 2 for Windows…..

• Database Consolidation Tips by Husnu Sensoy: you can get his presentation in his blog or here as local mirror
  

When called on an oracle database server it produces the following output:

host:~ # python mem.py
 Private  +   Shared  =  RAM used       Program
308.0 KiB +   0.0 KiB = 308.0 KiB       init
 84.0 KiB + 264.0 KiB = 348.0 KiB       zmd-bin
120.0 KiB + 284.0 KiB = 404.0 KiB       irqbalance
116.0 KiB + 404.0 KiB = 520.0 KiB       acpid
140.0 KiB + 400.0 KiB = 540.0 KiB       portmap
216.0 KiB + 428.0 KiB = 644.0 KiB       auditd
104.0 KiB + 544.0 KiB = 648.0 KiB       hald-addon-storage
364.0 KiB + 308.0 KiB = 672.0 KiB       klogd
232.0 KiB + 444.0 KiB = 676.0 KiB       wrapper
108.0 KiB + 572.0 KiB = 680.0 KiB       hald-addon-acpi
380.0 KiB + 304.0 KiB = 684.0 KiB       udevd
212.0 KiB + 508.0 KiB = 720.0 KiB       resmgrd
164.0 KiB + 592.0 KiB = 756.0 KiB       dd
424.0 KiB + 644.0 KiB =   1.0 MiB       slpd
576.0 KiB + 496.0 KiB =   1.0 MiB       dbus-daemon
332.0 KiB + 792.0 KiB =   1.1 MiB       cron (2)
584.0 KiB + 608.0 KiB =   1.2 MiB       mingetty (6)
508.0 KiB + 712.0 KiB =   1.2 MiB       nscd
384.0 KiB + 880.0 KiB =   1.2 MiB       vsftpd
276.0 KiB +   1.0 MiB =   1.3 MiB       sh
344.0 KiB +   1.1 MiB =   1.4 MiB       mysqld_safe
872.0 KiB + 660.0 KiB =   1.5 MiB       syslog-ng
856.0 KiB + 712.0 KiB =   1.5 MiB       ndo2db-3x (2)
820.0 KiB +   1.1 MiB =   1.9 MiB       powersaved
468.0 KiB +   1.6 MiB =   2.1 MiB       pickup
496.0 KiB +   1.7 MiB =   2.1 MiB       master
712.0 KiB +   1.6 MiB =   2.3 MiB       qmgr
  1.0 MiB +   1.3 MiB =   2.4 MiB       bash
720.0 KiB +   1.9 MiB =   2.6 MiB       smtpd
  2.0 MiB +   1.1 MiB =   3.0 MiB       mount.smbfs (2)
  2.3 MiB + 932.0 KiB =   3.2 MiB       hald
  1.4 MiB +   1.9 MiB =   3.3 MiB       sshd (2)
  1.4 MiB +   2.1 MiB =   3.5 MiB       smbd (2)
996.0 KiB +   2.6 MiB =   3.5 MiB       nagios
  3.1 MiB +   1.2 MiB =   4.3 MiB       cupsd
  2.9 MiB +   1.8 MiB =   4.7 MiB       ntpd
  5.7 MiB + 716.0 KiB =   6.4 MiB       perl
  7.7 MiB + 776.0 KiB =   8.5 MiB       named
  6.6 MiB +   5.9 MiB =  12.5 MiB       httpd2-prefork (11)
 11.9 MiB +   1.2 MiB =  13.1 MiB       slapd
  9.7 MiB +   4.4 MiB =  14.1 MiB       emagent
 10.1 MiB +   5.3 MiB =  15.5 MiB       tnslsnr (2)
 13.3 MiB +   4.9 MiB =  18.2 MiB       exp
 22.1 MiB + 516.0 KiB =  22.6 MiB       nmbd
103.8 MiB +   1.2 MiB = 105.0 MiB       mysqld-max
569.7 MiB +   9.4 MiB = 579.1 MiB       java (4)
  1.5 GiB +   3.4 GiB =   4.9 GiB       oracle (169)

Private  +   Shared  =  RAM used       Program
Warning: Shared memory is slightly over-estimated by this system
for each program, so totals are not reported.

You can see the running oracle programs sum up to 4.9 GB memory useage. Keep in mind several local running oracle instances are counted together here because summarization is being done by program!

One important part of installing the oracle clusterware stack is to ensure device names for OCR and Voting disks do not change (called “persistent naming” or “persistent binding”). Starting with 11g R2 OCR and Voting disks can also be stored in ASM which eliminates the need for configuring persistent device names because ASM automatically detects the disks and ASM does not rely on fixed device names.

This post is a small guide how to configure persistent device names for the OCR and voting disks when installing oracle clusterware 10g R2 and above.

The experiences outlined here are based on the white paper “Configuring udev and device mapper for Oracle RAC 10g Release 2 and 11g” from oracle available here.

Environment

This post assumes the following environment:

• storage attached via fibre channel
• Operating system: SLES 10/11 or Red Hat 4/5 or Oracle Enterprise Linux 5
• no custom multipathing software is used (e.g. Powerpath)

For configuring multipathing we will use the Linux native method  – “multipath”. Multipath offers features such as persistent device naming, failover, load balancing and much more. See the manpage for more information.

Setting up persistent device naming

In order to set up persistent device naming the following steps are required:

1. Determine the device WWN
2. Create or edit the file /etc/multipath.conf and assign a device aliases
3. Check
4. Write a script to correct device permissions on system boot

Step 1- Determine the device WWN

Running multipath for the first time will produce a list of available devices together with their WWNs. The WWN is a unique and life-long number for identifying the LUN. Therefore we can safely use the WWN as identifier.

For example:

host:/ # multipath -l
3600601609e61260002bb4dc9edd4de11  dm-1 GC,RAID 5
[size=100G][features=0][hwhandler=1 emc]
\_ round-robin 0 [prio=-1][active]
\_ 2:0:0:2 sdc        8:32  [active][undef]
\_ 3:0:0:2 sdg        8:35  [active][undef]

3600601607a702600a0851e9bedc4de11 dm-2 DGC,RAID 5
[size=1.0G][features=0][hwhandler=1 emc]
\_ round-robin 0 [prio=-1][active]
\_ 3:0:0:0 sdf        8:80  [active][undef]
\_ 2:0:0:0 sde        8:70  [active][undef]

Multipath detected two LUNs: one of 100 GB size and a second one with 1 GB size. In real world scenarios there will be more LUNs but for the article two LUNs are more than enough.

The multipath entry can be read like this:

# device wwn                         device created in /dev    device information
3600601609e61260002bb4dc9edd4de11    dm-1                      DGC,RAID 5

# disk size     features        used storage type handler
[size=100G]     [features=0]    [hwhandler=1 emc]

# balancing algorithm     priority       status
\_ round-robin 0          [prio=-1]      [active]
\_ 2:0:0:2 sdc        8:32  [active][undef]       #<-- first device
\_ 3:0:0:2 sdg        8:35  [active][undef]       #<---second device

The most important thing here is the WWN. You should write the WWN down as this is your identifier which needs to be put in /etc/multipath.conf

Step 2 – Create or edit /etc/multipath.conf

For the two devices shown above with WWN  3600601609e61260002bb4dc9edd4de11 and 3600601607a702600a0851e9bedc4de11 we create the persistent device binding:

multipaths {

multipath {
wwid 3600601607a702600a0851e9bedc4de11
alias ocr_san_a
path_grouping_policy group_by_prio
path_checker readsector0
path_selector "round-robin 0"
failback immediate
polling_interval 5
no_path_retry fail
}
multipath {
wwid 3600601609e61260002bb4dc9edd4de11
alias asmdisk001
path_grouping_policy group_by_prio
path_checker readsector0
path_selector "round-robin 0"
failback immediate
polling_interval 5
no_path_retry fail
}

I wont go too deep into the meaning of the parameters but here are the most important ones:

• ALIAS: the most important part of the configuration file; it specifies the device name for the disk to be created in /dev/mapper
• FAILBACK: specifies the amount of time in seconds after which a prior failed path will be considered “active” again
• POLLING_INTERVAL: how often (seconds) check the path and availability
• NO_PATH_RETRY: immediate fails a path if errors are detected and does not queue I/O (Note: Oracle whitepapers recommend to use a value of “10″ or even “20″ which will queue I/O in error cases thus “halting” the system for a short period of time. You have to test it yourself in your configuration. “fail” works best for me but may be different at your site / configuration).

For the meaning of the parameters refer to the manpage of “multipath.conf”.

Step 3 – Check

For the changes to become active running “/etc/init.d/multipath restart” is required. After that /dev/mapper should look like this:

host:/dev/mapper # ll
total 0
lrwxrwxrwx 1 root root          16 Nov 20 13:33 control -> ../device-mapper
brw-rw---- 1 asm  dba      253,  5 Nov 20 13:33 asmdisk001
brw-rw---- 1 crs  oinstall 253,  9 Nov 20 13:33 ocr_san_a

As you can see the disks are named according to the alias specified. If a disk does not appear check the log files or run “multipath -v 2″.

Note: I strongly recommend to create and use partitions on the configured devices regardless if they will be used for ASM, OCR or Voting Disks. In addition to that i recommend to configure aliases for asm disks as well.

If you created partions one partition on each device /dev/mapper will look like this:

host:/dev/mapper # ll
total 0
lrwxrwxrwx 1 root root          16 Nov 20 13:33 control -> ../device-mapper
brw-rw---- 1 asm  dba      253,  5 Nov 20 13:33 asmdisk001
brw-rw---- 1 asm  dba      253, 17 Nov 23 12:23 asmdisk001-part1
brw-rw---- 1 crs  oinstall 253,  9 Nov 20 13:33 ocr_san_a
brw-rw---- 1 crs  oinstall 253, 16 Nov 23 11:26 ocr_san_a-part1

Device “asmdisk001-part1″ is the first parition on device (disk) “asmdisk001″. For labeling the disk as ASM disk you specify “asmdisk001-part1″.

Step 4 – Write a script to correct device permissions

Oracle clusterware and asm require custom device permissions (the clusterware stack requires to own the OCR and voting disks and the ASM instance requires to own the ASM disks.) . By default disk devices are owned by user “root” and group “disk”. Unfortunately devices created in /dev/mapper cannot be influenced by udev rules (according to the whitepaper; i did not checked it myself). So i wrote a script to correct device permissions:

#! /bin/sh
#
# /etc/init.d/ocr_permission
#
### BEGIN INIT INFO
# Provides:          ocr_permission
# Required-Start:       $local_fs multipathd
# Should-Start:
# Required-Stop:
# Default-Start:    2 3 5
# Default-Stop:
# Description:       run ocr and voting disk permission corrections
### END INIT INFO

. /etc/rc.status
. /etc/sysconfig/sysctl

rc_reset

case "$1" in
 *)
 chown crs:oinstall /dev/mapper/ocr*
 chmod 660 /dev/mapper/ocr*

 chown crs:oinstall /dev/mapper/voting*
 chmod 660 /dev/mapper/voting*

 chown asm:dba /dev/mapper/disk*
 chmod 660 /dev/mapper/disk*
 ;;
esac

Note user “crs” holds the clusterware installation and user “asm” holds the ASM installation. Therefore OCR and voting disks will be owned by “crs” and asm disks will be owned by “asm”.

This script will be called upon system boot to correct the device permissions. It shall be started after the multipathing daemon. If you want to you can call this script periodically from cron to correct device permissions for devices added after system boot.

Version and component specific information

Use with Oracle Clusterware 10g R2

Oracle clusterware 10g R2 requires raw devices for OCR and voting disks.

To configure raw deivce mapping edit the file /etc/sysconfig/rawdevices (OEL/RedHat) or /etc/raw (SuSE) as follows:

# /etc/sysconfig/rawdevices
# SAMPLE FILE

raw1:/dev/mapper/ocr_sana-part1
raw2:/dev/mapper/ocrmirror_sanb-part1
raw3:/dev/mapper/voting1-part1
raw4:/dev/mapper/voting2-part1
raw5:/dev/mapper/voting3-part1

Restart raw service after changing the file to pick up the changes. Do not forget to add raw service to system boot sequence!

For RHEL5, OEL5, SLES10 to correct device permissions create a file /etc/udev/rules.d/99-raw.rules with the following content

KERNEL=="raw[1-2]*", GROUP="oinstall", MODE="640"
KERNEL=="raw[3-5]*", OWNER="crs", GROUP="oinstall", MODE="660"

For RHEL4, OEL4 or SLES9 refer to the oracle whitepaper mentioned above.

When installing clusterware stack OCR location would be according to our example: “/dev/raw/raw1″.

Use with Oracle Clusterware 11g R1

Note: Oracle itself recommends to use the latest clusterware and ASM software stack for new installations. I strongly recommend to install 11.1.0.7 clusterware and ASM stack (plus PSU if available) for new installations because installation is easier, software is more robust and many errors from 10g R2 are fixed (e.g. online addition of voting disks is possible as of 11g R1).

Starting with 11g R1 clusterware deprecated raw devices and can use block devices for OCR and voting disks directly. So beside configuring persistent device naming there is nothing more to do. When installing the clusterware based on this guide you would specify “/dev/mapper/ocr_san_a-part1″ for ocr location.

Use with Oracle Clusterware 11g R2

Starting with 11g R2 OCR and voting disks can be placed inside ASM. Theoretically this eliminates the need for persistent device binding / naming because ASM automatically detects the disks but i recommend to use device naming anyway as outlined below.

What about ASM?

Basically ASM does not require persistent device naming because ASM automatically detects devices and multipathing devices. Due to this you could label “/dev/dm-*” devices as ASM disks an use them at the disadvantage that these device names are not consistent across all nodes and can (and will) change.

So if you already configured persistent device naming for OCR and voting disks i recommend to do so as well for the ASM disks. The advantages are for instance more readable device names, granular configuration. Readble device names are important for normal or external redundancy configuration to distinguish the device from each other without reading ASM labels.

For normal redundancy configuration (two SAN storages) device mapping can look like this:

host:/dev/mapper # ll
total 0
lrwxrwxrwx 1 root root          16 Nov 20 13:33 control -> ../device-mapper
brw-rw---- 1 asm  dba      253,  0 Nov 20 13:33 disk001_sana
brw-rw---- 1 asm  dba      253, 11 Nov 23 12:41 disk001_sana-part1
brw-rw---- 1 asm  dba      253,  5 Nov 20 13:33 disk001_sanb
brw-rw---- 1 asm  dba      253, 18 Nov 23 12:41 disk001_sanb-part1
brw-rw---- 1 asm  dba      253,  1 Nov 20 13:33 disk002_sana
brw-rw---- 1 asm  dba      253, 19 Nov 23 12:41 disk002_sana-part1
brw-rw---- 1 asm  dba      253,  7 Nov 20 13:33 disk002_sanb
brw-rw---- 1 asm  dba      253, 15 Nov 23 12:41 disk002_sanb-part1
brw-rw---- 1 crs  oinstall 253,  4 Nov 20 13:33 ocr_san_a
brw-rw---- 1 crs  oinstall 253, 13 Nov 23 12:07 ocr_san_a-part1
brw-rw---- 1 crs  oinstall 253,  8 Nov 20 13:33 ocrmirror_san_b
brw-rw---- 1 crs  oinstall 253, 10 Nov 23 12:07 ocrmirror_san_b-part1
brw-rw---- 1 crs  oinstall 253,  3 Nov 20 13:33 voting1_san_a
brw-rw---- 1 crs  oinstall 253, 16 Nov 23 12:41 voting1_san_a-part1
brw-rw---- 1 crs  oinstall 253,  9 Nov 20 13:33 voting2_san_b
brw-rw---- 1 crs  oinstall 253, 12 Nov 23 12:41 voting2_san_b-part1

The customer requested the installation of an 2-node-rac cluster running a 10g Release 2 database. Storage was attached via fibre channel coming from two EMC AX25 storage systems directly attached to both nodes. The cluster was installed with  11.1.0.7.0 clusterware and 11.1.0.7.0 asm. The database to be run was 10.2.0.4.2. By using ASM and normal redundancy mode both storage arrays were mirrored against each other. Everything worked well – too well. So after installing the whole cluster and setting up the database instance we performed some tests:

The first test was to interrupt the connection (pull the cable!) between storage array A and node A. From my experiences with 11g R2 and asm in normal redundancy mode i expected to database to stay up and running. To my surprise the database on node A crashed. In addition to that i was unable to start the instance again. I cannot give the exact error message because the project ended and i am not allowed to disclose the messages. Among several ORA-00600 messages i also saw messages saying the database was unable to write to the control file and open the redo logs. That was strange because ASM already started to dropping the missing disks from the disk group. In addition dropping the missing disks was not that easy. After re-adding the disks back to the disk group i tried to delete one LUN from the operating system selectively thus keeping communication, links and so on intact. The result did not change: The database instance on the affected node crashed and i was unable to start the instance again. After searching at meta link which yielded nothing we decided to give 11.1.0.7.1 a try:

After installing and creating a 11.1.0.7.1 database and increasing disk group compatibility from 10.2 to 11.1 we tried again to interrupt the connection between one storage array and one node. This time the instance crashed with an I/O error and was terminated by the log writer but was restarted seconds after that fine -  a great improvement over 10.2. After discussion the customer decided to go with 11.1.0.7.1 instead of 10.2.0.4.2 and we continued our tests which were completed successfully. These tests involved for instance interrupting the communication from one storage array to both nodes and re-establishing the connection again (after waiting 15 minutes).

My conclusion is: ASM in mirrored configuration can be used from 11g Release 1 onwards and gets well with 11g R2 (see my tests with 11g R2 and ASM in normal redundancy configuration). With 10g ASM should rely on external redundancy.

As always: Please comment! I´m looking forward from your experiences with ASM in normal or high redundancy configurations and 10g R2 or 11g R1 databases.

“can we change kernel parameter in linux when oracle is running”

The answer is: Yes, of course!

You can alter any kernel parameter as long as it don’t require a reboot. If you set any kernel parameter to a lower value than your running database requires there wont be any impact in time – but there might be problems when your database continues to run (e.g. unable to start new sessions, degraded performance).Once your database is shut down and you lowered for instance the SHMMAX parameter you wont be able to start the instance.

Therefore you should make sure to increase the values and check twice!