While working on my most recent project i tested an interesting ZFS feature: writeable snapshots aka “clones”.
While testing i came to the conclusion that with the help of clones oracle databases (and, of course – other databases as well) can be clones extremely fast with almost no disk space required. So i tested it.
Read on for the results…
This is part III of my most recent project named “Building a custom and cheap storage server yourself”. Part I can be found here and Part II here.
Part II left with a current status of a crashing machine during high I/O loads.During the previous weeks i tried to solve the stability problems. And finally: we solved them!
This third part is about our problems, what caused them and how we finally solved them….
Just a short note from myself:
Last week i successfully completed all requirements to become an
Oracle Certified Trainer
Preparing for the course took all my time so there was little time left to write in my blog. Beside preparing for the trainer tests i worked a lot on my other project: building a cheap custom storage yourself.
It´s been a few months since posting the idea of building a custom made storage system in my blog. During this time we convinced people to give our idea a try, did some minor changes to the box layout and ordered the parts. Finally they arrived at 27th December 2009.
So this is part II of the project called “Building a custom and cheap storage server yourself”. For people dont know what i am talking about: part I can be found here and part III here.
We had a budget of approx 12.000 euros for building the prototype of the storage box. We decided to build the prototype with 40 disks from the start and fit an SAN-HBA in it to try COMSTAR (this enables us to export the storage via SAN to other servers). As operating system we choose Open Solaris. Two disks are dedicated to the operating system and are attached directly to the mainboard. The remaining 38 disks are attached to either one Adaptec 52445 or one Adaptec 51645 controller.
Below are some pictures of the components and the final assembled box. Click on the image for a larger version:
When checking OTN today i noticed the release of Oracle 11g Release 2 (11.2.0.1.0) for
I am surprised 11g R2 for Itanium was released prior 11g Release 2 for Windows…..
Perhaps a little bit late but this post is a catch-up of interesting speeches held at the recent Oracle open world 2009:
Sometimes administrators want to know how much memory is being used by a program (and not per process). This is especially useful to calculate the memory consumption for an oracle instance. Today i found a nice script for doing so here (local mirror).
When called on an oracle database server it produces the following output:
host:~ # python mem.py Private + Shared = RAM used Program 308.0 KiB + 0.0 KiB = 308.0 KiB init 84.0 KiB + 264.0 KiB = 348.0 KiB zmd-bin 120.0 KiB + 284.0 KiB = 404.0 KiB irqbalance 116.0 KiB + 404.0 KiB = 520.0 KiB acpid 140.0 KiB + 400.0 KiB = 540.0 KiB portmap 216.0 KiB + 428.0 KiB = 644.0 KiB auditd 104.0 KiB + 544.0 KiB = 648.0 KiB hald-addon-storage 364.0 KiB + 308.0 KiB = 672.0 KiB klogd 232.0 KiB + 444.0 KiB = 676.0 KiB wrapper 108.0 KiB + 572.0 KiB = 680.0 KiB hald-addon-acpi 380.0 KiB + 304.0 KiB = 684.0 KiB udevd 212.0 KiB + 508.0 KiB = 720.0 KiB resmgrd 164.0 KiB + 592.0 KiB = 756.0 KiB dd 424.0 KiB + 644.0 KiB = 1.0 MiB slpd 576.0 KiB + 496.0 KiB = 1.0 MiB dbus-daemon 332.0 KiB + 792.0 KiB = 1.1 MiB cron (2) 584.0 KiB + 608.0 KiB = 1.2 MiB mingetty (6) 508.0 KiB + 712.0 KiB = 1.2 MiB nscd 384.0 KiB + 880.0 KiB = 1.2 MiB vsftpd 276.0 KiB + 1.0 MiB = 1.3 MiB sh 344.0 KiB + 1.1 MiB = 1.4 MiB mysqld_safe 872.0 KiB + 660.0 KiB = 1.5 MiB syslog-ng 856.0 KiB + 712.0 KiB = 1.5 MiB ndo2db-3x (2) 820.0 KiB + 1.1 MiB = 1.9 MiB powersaved 468.0 KiB + 1.6 MiB = 2.1 MiB pickup 496.0 KiB + 1.7 MiB = 2.1 MiB master 712.0 KiB + 1.6 MiB = 2.3 MiB qmgr 1.0 MiB + 1.3 MiB = 2.4 MiB bash 720.0 KiB + 1.9 MiB = 2.6 MiB smtpd 2.0 MiB + 1.1 MiB = 3.0 MiB mount.smbfs (2) 2.3 MiB + 932.0 KiB = 3.2 MiB hald 1.4 MiB + 1.9 MiB = 3.3 MiB sshd (2) 1.4 MiB + 2.1 MiB = 3.5 MiB smbd (2) 996.0 KiB + 2.6 MiB = 3.5 MiB nagios 3.1 MiB + 1.2 MiB = 4.3 MiB cupsd 2.9 MiB + 1.8 MiB = 4.7 MiB ntpd 5.7 MiB + 716.0 KiB = 6.4 MiB perl 7.7 MiB + 776.0 KiB = 8.5 MiB named 6.6 MiB + 5.9 MiB = 12.5 MiB httpd2-prefork (11) 11.9 MiB + 1.2 MiB = 13.1 MiB slapd 9.7 MiB + 4.4 MiB = 14.1 MiB emagent 10.1 MiB + 5.3 MiB = 15.5 MiB tnslsnr (2) 13.3 MiB + 4.9 MiB = 18.2 MiB exp 22.1 MiB + 516.0 KiB = 22.6 MiB nmbd 103.8 MiB + 1.2 MiB = 105.0 MiB mysqld-max 569.7 MiB + 9.4 MiB = 579.1 MiB java (4) 1.5 GiB + 3.4 GiB = 4.9 GiB oracle (169) Private + Shared = RAM used Program Warning: Shared memory is slightly over-estimated by this system for each program, so totals are not reported.
You can see the running oracle programs sum up to 4.9 GB memory useage. Keep in mind several local running oracle instances are counted together here because summarization is being done by program!
Installing oracle clusterware has been made easier and easier from release to release. When installing 10g R1 you struggled with ssh equivalence, wrong directory permissions and script bugs was a pain. 10g R2 was better but still quite painful (especially the step from 10.2.0.3.0 to 10.2.0.4.0). With 11g R1 installing the clusterware was made fairly robust and was even more simplified with 11g R2.
One important part of installing the oracle clusterware stack is to ensure device names for OCR and Voting disks do not change (called “persistent naming” or “persistent binding”). Starting with 11g R2 OCR and Voting disks can also be stored in ASM which eliminates the need for configuring persistent device names because ASM automatically detects the disks and ASM does not rely on fixed device names.
This post is a small guide how to configure persistent device names for the OCR and voting disks when installing oracle clusterware 10g R2 and above.
The experiences outlined here are based on the white paper “Configuring udev and device mapper for Oracle RAC 10g Release 2 and 11g” from oracle available here.
It´s been a while since my last post because i was busy doing some projects. One of these projects involved installing a rac cluster with ASM in normal redundancy mode. My experiences installing this configuration is covered in this article.
The customer requested the installation of an 2-node-rac cluster running a 10g Release 2 database. Storage was attached via fibre channel coming from two EMC AX25 storage systems directly attached to both nodes. The cluster was installed with 11.1.0.7.0 clusterware and 11.1.0.7.0 asm. The database to be run was 10.2.0.4.2. By using ASM and normal redundancy mode both storage arrays were mirrored against each other. Everything worked well – too well. So after installing the whole cluster and setting up the database instance we performed some tests:
The first test was to interrupt the connection (pull the cable!) between storage array A and node A. From my experiences with 11g R2 and asm in normal redundancy mode i expected to database to stay up and running. To my surprise the database on node A crashed. In addition to that i was unable to start the instance again. I cannot give the exact error message because the project ended and i am not allowed to disclose the messages. Among several ORA-00600 messages i also saw messages saying the database was unable to write to the control file and open the redo logs. That was strange because ASM already started to dropping the missing disks from the disk group. In addition dropping the missing disks was not that easy. After re-adding the disks back to the disk group i tried to delete one LUN from the operating system selectively thus keeping communication, links and so on intact. The result did not change: The database instance on the affected node crashed and i was unable to start the instance again. After searching at meta link which yielded nothing we decided to give 11.1.0.7.1 a try:
After installing and creating a 11.1.0.7.1 database and increasing disk group compatibility from 10.2 to 11.1 we tried again to interrupt the connection between one storage array and one node. This time the instance crashed with an I/O error and was terminated by the log writer but was restarted seconds after that fine - a great improvement over 10.2. After discussion the customer decided to go with 11.1.0.7.1 instead of 10.2.0.4.2 and we continued our tests which were completed successfully. These tests involved for instance interrupting the communication from one storage array to both nodes and re-establishing the connection again (after waiting 15 minutes).
My conclusion is: ASM in mirrored configuration can be used from 11g Release 1 onwards and gets well with 11g R2 (see my tests with 11g R2 and ASM in normal redundancy configuration). With 10g ASM should rely on external redundancy.
As always: Please comment! I´m looking forward from your experiences with ASM in normal or high redundancy configurations and 10g R2 or 11g R1 databases.
I found the following question in my blog statistics:
“can we change kernel parameter in linux when oracle is running”
The answer is: Yes, of course!
You can alter any kernel parameter as long as it don’t require a reboot. If you set any kernel parameter to a lower value than your running database requires there wont be any impact in time – but there might be problems when your database continues to run (e.g. unable to start new sessions, degraded performance).Once your database is shut down and you lowered for instance the SHMMAX parameter you wont be able to start the instance.
Therefore you should make sure to increase the values and check twice!
