Part II left with a current status of a crashing machine during high I/O loads.During the previous weeks i tried to solve the stability problems. And finally: we solved them!

This third part is about our problems, what caused them and how we finally solved them….

Problems, Problems

Part I – using Adaptec Controller

Initially we equipped our storage with one Adaptec 52445 or one Adaptec 51645. These adapters were chosen because they offered a large amount of SATA ports (16 and 24) so we could attach all remaining disks directly without any expander.

During our tests we noticed a compete halt of all I/O under high I/O load. Not even a “ls” returns any result. The problem was discussed at the opensolaris mailing list but without any insight. Even the message log did not showed any error message.

I tried several settings on controller side, including creating a RAID array, turning off NCQ, setting timeout values higher and so on. But nothing worked. Under load sooner or later the I/O to all drives attached to the adaptec controller came to a complete halt.

After noticing that high I/O to disks directly attached to the motherboard do not “lock up” in any way we were pretty sure out problems were caused by the adaptec controller…. so we replaced them.

Part II – using LSI 1086E-based controllers

In order to solve our problems which were most probably related to the adaptec controllers we replaced them with two LSI 1086E-based controllers plus one expander attached to each controller. We used the Chenbro CK12804 expander.

Unfortunately we observed the same behavior with these controllers as well: the I/O stalled sooner or later.

So i took a look at the hard disks next. For the prototype we ordered fourty Seagate ST31000340NS disks with 1 TB each. Note that these disks are – according to the seagate homepage – designed for:

"The Barracuda ES.2 drive is the perfect solution for high-capacity enterprise storage
applications such as the migration of mission-critical transactional data, from tier 1
to tier 2 (nearline) storage, where dollars/GB and GB/watt are a primary concern."

Another criteria for chosing this drive was:

"- 24x7 operation and 1.2 M hrs. MTBF"

After making sure we used the most recent firmware (actually “SN06″ is the most recent firmware) i took a look at the Seagate forums and found an interesting thread about Seagate ES.2 drives (the drive we used) and some firmware problems. Especially one post took my attention:

I've got a setup with two 1TB disks which previously used the AN05 firmware and they
worked just fine, no matter how many GB's of data I copied to/from the RAID1 mirror.
Right after updating the firmware to suggested SN06 version it seems that I must
disable NCQ completely or the system will hang in 30 mins if there are any data being
copied from the disks. Because there has not been any guarantees that AN05-version of
the firmware doesn't suffer from the disk locking bug, I did not feel comfortable in
continuing using that version even though it seemed to work just fine.

Well, i thought to myself: “You´ve already tried disabling NCQ at /etc/system level and controller level with the adaptec controller but anyway – try it!”. So i turned off NCQ at LSI controllers with the help of LSIutil.

After booting the system again i put some I/O load on it and waited for the problem to appear again…. after waiting pessimistically for days it did not appear. So i once again booted the system and increased the I/O pressure and waited again. Soon after some errors were shown:

Jan 29 12:00:48 openstorage scsi: [ID 107833 kern.notice]
scsi_state = 0, transfer count = 1400, scsi_status = 0
Jan 29 12:00:48 openstorage scsi: [ID 107833 kern.warning] WARNING:
/pci@0,0/pci8086,340c@5/pci1014,396@0/sd@a,0 (sd52):
Jan 29 12:00:48 openstorage     incomplete read- retrying
Jan 29 12:00:48 openstorage scsi: [ID 107833 kern.warning] WARNING:
/pci@0,0/pci8086,340c@5/pci1014,396@0 (mpt1):
Jan 29 12:00:48 openstorage     unknown ioc_status = 4
Jan 29 12:00:48 openstorage scsi: [ID 107833 kern.notice]
scsi_state = 0, transfer count = 12800, scsi_status = 0
Jan 29 12:00:48 openstorage scsi: [ID 107833 kern.warning] WARNING:
/pci@0,0/pci8086,340c@5/pci1014,396@0/sd@8,0 (sd50):
Jan 29 12:00:48 openstorage     incomplete read- retrying

Despite of these errors a complete lockup of all I/O was not observed anymore. So disabling NCQ did the trick as the following days (and weeks) proved.

Obviously Seagates SATA “server” disks have some kind of problem with NCQ. Using an older firmware was not an option because older firmware versions might cause the disks to fail completely. A newer firmware than SN06 was also not available. I wont comment this any further but most likely i will think twice before buying any Seagate disks again. I even contacted Seagate support but i have not heared anything from them.

Interestingly disabling NCQ at the Adaptec controller and even on /etc/system level did not work. I have not investigated this further.

Part III – getting rid of “incomplete read- retrying” messages

After replacing the adaptec controller with some LSI controller and disabling NCQ for all disks the system stabilized. Even extremely high I/O would not lock up anymore.

But there were still error messages like this:

Jan 29 12:00:48 openstorage     incomplete read- retrying
Jan 29 12:00:48 openstorage scsi: [ID 107833 kern.warning] WARNING:
/pci@0,0/pci8086,340c@5/pci1014,396@0 (mpt1):
Jan 29 12:00:48 openstorage     unknown ioc_status = 4

When debugging these errors i noticed every time an “incomplete read- retrying” message was issued the error counters on the physical link level was increased:

Adapter Phy 6:  Link Up
 Invalid DWord Count                                      1
 Running Disparity Error Count                            1
 Loss of DWord Synch Count                                0
 Phy Reset Problem Count                                  0

<INCOMPLETE READ ERROR MESSAGE>

Adapter Phy 6:  Link Up
 Invalid DWord Count                                      2
 Running Disparity Error Count                            2
 Loss of DWord Synch Count                                0
 Phy Reset Problem Count                                  0

So it seemed related either to the physical connection – or – still the hard disks. Before digging further on hard disk level we replaced the cables with new ones. The error persisted.

In the thread quoted above the user not only reported problems with NCQ but changed the SATA interface speed from 3 gbit/s to 1.5 gbit/s as well. According to his findings changing the speed is not a viable workaround. But i tried it anyway. After chaning the speed of the drives interface to 1.5 Gbit/s the error disappeared as well.

The controller configuration looks like this:

Main menu, select an option:  [1-99 or e/p/w or 0 to quit] 16

SAS1068E's links are 1.5 G, 1.5 G, 1.5 G, 1.5 G, 3.0 G, 3.0 G, 3.0 G, down
 B___T     SASAddress     PhyNum  Handle  Parent  Type
        500605b0017ff110           0001           SAS Initiator
        500605b0017ff111           0002           SAS Initiator
        500605b0017ff112           0003           SAS Initiator
        500605b0017ff113           0004           SAS Initiator
        500605b0017ff114           0005           SAS Initiator
        500605b0017ff115           0006           SAS Initiator
        500605b0017ff116           0007           SAS Initiator
        500605b0017ff117           0008           SAS Initiator
 0   5  09221b095e5c7c67     0     0009    0001   SATA Target
 0   6  09221b095d6f556b     1     000a    0002   SATA Target
 0   7  09221b087d585275     2     000b    0003   SATA Target
 0   8  09221b087a5a7a6c     3     000c    0004   SATA Target
        5001c450000c4700     4     000d    0005   Edge Expander
 0   9  5001c450000c470c    12     000e    000d   SATA Target
 0  10  5001c450000c470d    13     000f    000d   SATA Target
 0  11  5001c450000c470e    14     0010    000d   SATA Target
 0  12  5001c450000c470f    15     0011    000d   SATA Target
 0  13  5001c450000c4710    16     0012    000d   SATA Target
 0  14  5001c450000c4711    17     0013    000d   SATA Target
 0  15  5001c450000c4712    18     0014    000d   SATA Target
 0  16  5001c450000c4713    19     0015    000d   SATA Target
 0  17  5001c450000c4714    20     0016    000d   SATA Target
 0  18  5001c450000c4715    21     0017    000d   SATA Target
 0  19  5001c450000c4716    22     0018    000d   SATA Target
 0  20  5001c450000c4717    23     0019    000d   SATA Target
 0  21  5001c450000c4718    24     001a    000d   SATA Target
 0  22  5001c450000c4719    25     001b    000d   SATA Target
 0  23  5001c450000c471a    26     001c    000d   SATA Target
 0  24  5001c450000c471b    27     001d    000d   SATA Target
 0  25  5001c450000c473d    28     001e    000d   SAS Initiator and Target

Type      NumPhys    PhyNum  Handle     PhyNum  Handle  Port  Speed
Adapter      8          0     0001  -->    0     0009     0    1.5
  1     0002  -->    0     000a     1    1.5
  2     0003  -->    0     000b     2    1.5
  3     0004  -->    0     000c     3    1.5
  4     0005  -->    0     000d     4    3.0
  5     0005  -->    1     000d     4    3.0
  6     0005  -->    2     000d     4    3.0
Expander    30          0     000d  -->    4     0005     4    3.0
  1     000d  -->    5     0005     4    3.0
  2     000d  -->    6     0005     4    3.0
 12     000d  -->    0     000e     4    1.5
 13     000d  -->    0     000f     4    1.5
 14     000d  -->    0     0010     4    1.5
 15     000d  -->    0     0011     4    1.5
 16     000d  -->    0     0012     4    1.5
 17     000d  -->    0     0013     4    1.5
 18     000d  -->    0     0014     4    1.5
 19     000d  -->    0     0015     4    1.5
 20     000d  -->    0     0016     4    1.5
 21     000d  -->    0     0017     4    1.5
 22     000d  -->    0     0018     4    1.5
 23     000d  -->    0     0019     4    1.5
 24     000d  -->    0     001a     4    1.5
 25     000d  -->    0     001b     4    1.5
 26     000d  -->    0     001c     4    1.5
 27     000d  -->    0     001d     4    1.5
 28     000d  -->    0     001e     4    3.0

As you can see the connection Controller –> Expander has a bandwidth of 3.0 Gbit/s whereas Expander –> Drive is limited to 1.5 Gbit/s.

Conclusion

After disabling NCQ on controller lever and limiting the bandwidth between expander and hard disk to 1.5 Gbit/s for each disk all problems are gone.

All? Yes, indeed. For three weeks there are NO error messages anymore and the system itself is rock-stable. During that period one hard disk physically failed and had to be replaced (remember, i wont comment on the quality of these “server” hard disks….).

Tomorrow or the day after tomorrow i wil post Part IV covering first performance benchmarks.

So this is part II of the project called “Building a custom and cheap storage server yourself”. For people dont know what i am talking about: part I can be found here and part III here.

Building the box

We had a budget of approx 12.000 euros for building the prototype of the storage box. We decided to build the prototype with 40 disks from the start and fit an SAN-HBA in it to try COMSTAR (this enables us to export the storage via SAN to other servers). As operating system we choose Open Solaris. Two disks are dedicated to the operating system and are attached directly to the mainboard. The remaining 38 disks are attached to either one Adaptec 52445 or one Adaptec 51645 controller.

Below are some pictures of the components and the final assembled box. Click on the image for a larger version:

Installing

After putting everything in place we started to install Open Solaris for the first time. But wait – we forgot a DVD drive. But USB-attached dvd drives are widely available. After attaching a usb-dvd drive we started to install.

This time we were able to boot from dvd but installation did not start. A short glance at the mainboard manufacturer site yielded:

Fixed an issue where the system could not boot from a DVD ROM when an Adaptec Raid Card
(ASR-5805) was installed on any PCI-e Slot

Yup, we had a similar controller installed. So – instead of shorthand installing the operating system and playing around a little bit – we patched all components to the most recent version (BIOS, HBA, and so on).

After doing so installation went fine and the system booted Open Solaris 2009.06 for the first time.

Patching (again)

Open Solaris 2009.06 is quite old and we needed the latest features of ZFS and COMSTAR so we upgraded our Open Solaris. There are basically two versions available:

• A “release” version (currently as of December 2009: Build 111) of Open Solaris (Repository Link)
• A “development” version (currently as of December 2009: Build 130) of Open Solaris (Repository Link)

The upgrade process is pretty straight forward:

Upgrading to the most recent “release” version is done by entering:

pfexec pkg image-update

Upgrading to the most recent “development” build is done by:

$ pfexec pkg set-publisher -O http://pkg.opensolaris.org/dev opensolaris.org
$ pfexec pkg image-update

We first patched to the most recent “release” build and afterwards patched to the most recent “developement” build.Beside from the painfully slow fetching of the packages (*really* slow… it run several hours for few hundred MB) everything worked fine and we ended up with two bootable configurations:

• Release – Build 111
• Development – Build 130

We continued our work with the development build due to the fact we wanted deduplication and the most recent version with the most recent fixes included.

Configure

Replacing the Adaptec controller driver

In order to use the drives attached to the Adaptec controller we had to replace the driver shipped with Open Solaris with the appropriate driver from Adaptec.

Creating the ZFS pool

After successfully booting with Build 130 and replacing the adaptec driver we created our first ZFS pool “pool1″ with a total capacity of 20 TB:

zpool create pool1 raid2z c10t0d0s0 c11t0d0s0 c12t0d0s0 c13t0d0s0 c14t0d0s0 c15t0d0s0 c16t0d0s0 c17t0d0s0
c18t0d0s0 c19t0d0s0 c20t0d0s0 c21t0d0s0 c22t0d0s0 c23t0d0s0 c24t0d0s0 c25t0d0s0 c26t0d0s0 c27t0d0s0
c28t0d0s0 c29t0d0s0 c30t0d0s0 c31t0d0s0

After several seconds the zpool was created successfully. So far – so cool.

I know a raid1z pool with 20 disks i pretty uncommon and i would never ever use this configuration in production due to the extremely high probability of a double-disk-failure. But for the very first tests it seemed acceptable.

Install missing packages

For using COMSTAR (and iSCSI as well) we needed several packages which could be installed with the help of the package manager (“pkg”) easily.

Booting after COMSTAR and ZPool creation

The first boot after installing the COMSTAR packages, replacing the qlc with the qlt driver (to export our storage over SAN) and creating the large zpool caused the system to crash during boot.

It took me some time to find out package management silently uninstalled the adaptec controller driver from adaptec we installed before and replaced it with the original Open Solaris driver (which cannot use disks attached to the adaptec controller) when installing the COMSTAR packages. Booting afterwards caused the system to crash.

Testing and Crashing

Currently we are testing overall system performance. While doing so we faced problems were I/O to the adaptec controller would hang occasionally. The system is still responsible (most probably because the system disks are not attached via the adaptec controller) but I/O to the data pool is impossible. I´ve posted the problem at the opensolaris mailing list but currently with no replies.

Conclusion: currently crashing

Although we are facing some problems we will continue this project. First of all we separated the hard disks on each controller into their own zpools. If this is a hardware issue the error should be located to one zpool only. In addition to that we will replace the controller with LSI-based ones which are also used in the SUN Thumper systems.

If the system runs performance is good: We observed up to 1 GB/s (not Gbit!) or 1000 MB/s sequential read/write speed and up to 7500 I/O operations per second. Exporting storage via SAN works as well with decent speed: We observed up to 320 MB/s on a QLogic 4 Gbit/s HBA running under Linux SLES 10 SP3.

A colleague of me and i saw this project and wondered if this kind of storage could be used for databases as well. So we analyzed the design and noticed some problems from our point of view:

• data access only via HTTP
• they used the JFS file system which is not widely used
• generally hard disks are hot-swappable; but not used due to fear of problems
• optimized for space rather than for speed
• relatively “weak” power supply

So we tried to improve the layout with the following constraints:

• approx. 10.000 Euro (approx. 15.000 US-Dollar) in total
• Storage accessible via multiple protocols:
  • NFS
  • iSCSI
  • NFS
  • CIFS
  • if possible SAN
• Reliable
• Optimized for speed rather than capacity (remember: we talked about databases)
• Hot-Swapable Harddisks

This is part I of our journey towards building a storage system ourself. Part II is here and Part III here.

Required Components

What components do we need?

• Case
• Power Supply
• Motherboard
• CPU
• Memory
• Storage HBAs for the hard disks (SAS/SATA)
• SAS/SATA Expander if needed
• Network Interface Cards (1 / 10 Gbit/s)
• SAN HBA
• Operating System

Case

Cases in several sized are available by several vendors. Most of them are built in Asia and sold by resellers all over the world.

For supporting hot swapping hard disks the case needed to support this as well. So we looked at several cases and came up with this case here:

• up to 42 hot swappable hard disks
• three redundant power supplies
• room for a quad cpu board
• Backplane
  • ten backplanes connecting four drives each
  • backplane connected by SFF 8087 (aka. “mini-SAS”) cables with controller
  • separate SFF8087 cable required for connecting system drives with controller
  • so for connecting all 42 disks (40 + 2) you need 11 SFF8087 cables

We are running this project in Germany so we choose a German vendor. Searching at Google for the case’s name (“RSC-8ED-0Q1″)  yield you can buy the case at several vendors, for instance here.

Some pictures and technical data sheet:

Mainboard

According to the data sheet there are two system boards available. We have chosen the Tyan board with the following technical specifications:

• integrated graphic controller
• Slots
  • 2x PCI-E x16; x16 sig.
  • 2x PCI-E x16; x4 sig.
  • 1 PCI 32-bit
• 8 SATA-2-Ports
• 2x USB 2.0
• 3x 1 Gbit/s-NIC
• 4x 1207-pin socket for AMD Opteron (Rev. F) 8000er CPUs
• 16x DDR2-DIMMS (max. 64 GB RAM)

The costs for the system board are approx 250 euros or 368 US-$.

Memory

• system board supports up to 16 DDR2 DIMM Modules
• up to 64 GB Memory

Due to the current low price of memory we decided to put the largest allowed amount of memory (64 GB) at the system board.

One 4 GB memory module costs approx. 80 euros (or 117.8 US-$). So sixteen of these modules add up to 1280 euros or 1885 US-$.

CPU

The board imposes some restrictions on the type of CPU we can use. We decided to fit four quad core CPUs. The CPU chosen was:

• AMD Opteron 2350 (Sockel F, 65nm, Barcelona, OS2350WAL4BGHWOF)
• Socket F
• Clock Frequency: 2.000 MHz
• Quad-Core
• Type of core: Barcelona
• Stepping (Revision): B3-Stepping
• FSB: 1.000 MHz
• QuickPath Interfac
• Hypertransport: 2,0 GT/s
• Second-Level-Cache: 4 x 512 KB

250 Euros (or 368.2 US-$) for one CPU or 1000 euros (or 1472.75 US-$) for four CPUs.

SAS/SATA HBA and Expander Cards

We needed to attach 42 hard disks to the system board. Two disks are attached with a single mini-SAS cable. The remaining fourty disks are attached by ten mini-SAS cables (each cable attaching four disks). So we had to attach a total of eleven mini-SAS (SFF8087) cables.

SAS Cabling

• two disk connected over a single mini-SAS cable directly to the system board
• 20 disks connected over 5 mini-SAS cables to sas-expander A
  • sas-expander A connected over one mini-SAS cable to sas-controller A
• 20 disks connected over 5 mini-SAS cables to sas-expander B
  • sas-expander B connected over one min-SAS cable to sas-controller B

Storage Layout

Summarizing our cabling layout we have three different failure components from storage point of view:

• sas controller on system board
• sas controller A
• sas controller B

We bear this in mind when designing raid groups lateron.

Components

• 2x SAS Controller: Adaptec 2405 with the following features according to the documentation:
  • 128 MB Cache
  • Supports 4 direct-attached or up to 128 SATA or SAS disk drives using SAS expanders
  • Quick initialization
  • Online Capacity Expansion
  • Copyback Hot Spare
  • Dynamic caching algorithm
  • Native Command Queuing (NCQ)
  • Background initialization
  • Hot-plug drive support
  • RAID Level Migration
  • Hot spares – global, dedicated, and pooled
  • Automatic/manual rebuild of hot spares
  • SAF-TE enclosure management
  • Configurable stripe size
  • S.M.A.R.T. support
  • Multiple arrays per disk drive
  • Bad stripe table
  • Dynamic sector repair
  • Staggered drive spin-up
  • Bootable array support
  • Optimized Disk Utilization
  • Controller bietet RAID0, RAID1 und RAID10 in „Hardware“; nutzen wir aber nicht, da wir ZFS nutzen

• 2x SAS Expander: CHENBRO Low Profile 28-port SAS expander card
  • Input: up to 6 mini-SAS cables coming from the back plane
  • Output: one mini-SAS cable to the raid controller

NICs (1 Gbit/s)

Three NICs already on-board so nothing needs to be done here.

When building the system we have to measure the network performance of these interfaces. If performance is too bad we need to replace the on-board NICs with dedicated NICs for instance from Intel. When doing so the price for 10 GE-NICs should be checked again.

NICs (10 Gbit/s)

Due to lack of support in the core switches we did not embedded a 10 GE-NIC.

If you want to do so the following card will fit:

• Intel 10 GE XF SR NIC PCI-E
• Price: 2300 euro or 3387 US-$

SAN HBAs

Due to lack of a SAN environment we did not add this feature. But if you want to you can turn your storage system into a SAN target (i.e. exports storage) you can do this with the COMSTAR project whipping with Open Solaris.

For a impressing demonstration you can refer here.

For enabling SAN features you need a QLogic (not EMulex) HBA. I recommend two Single-Port HBAs. One single-port HBA costs approx. 500 Euros.

Operating system

A main goal for our storage project were minimal costs. So paying for an operating system was not desired. In addition to that freely available operating system like *BSD, Linux or Solaris are extremely stable, ship with a lot of features and mostly perform much better than commercial products.

We tested several operating systems and finally ended up with the OpenSolaris Project for many resons:

• extremely stable
• updated on regular basis
• wide protocol support available (iSCSI, FCoE, NFS, CIFS, HTTP, …)
• the ZFS file system
• the COMSTAR project

The last two features are the main reasons for chosing OpenSolaris. ZFS is a highly integrated and extremely powerful and flexible file system. Perhaps the most powerful file system currently available while the COMSTAR project enables us to export our storage over SAN.

Storage Layout

Based on the hardware and software configuration we decided not to use the RAID features of our SAS-controller and use ZFS with RAID-DP (RAID with Double Parity) instead. Because parity calculations are cpu resource intensive we fitted four quad-core-cpus.

Due to our cabling we have three main failure components:

• the internal (on-board) storage controller
• sas controller A in PCI-e-Slot
• sas controller B in PCI-e-Slot

For keeping the failure groups separated we designed the following raid configuration:

• two disk attached to the on-board controller:
  • one two-way-mirror for the operating system
• twenty disks attached via an sas expander card to sas raid controller A
  • one raid group consisting of 20 disks with one Hot Spare, two Parity Disks and 17 data disks
• twenty disk attached via an sas expander card to sas controller B
  • one raid group consisting of 20 disks with one Hot Spare, two Parity Disks and 17 data disks

Hard disks

In our calculation there is one component left: The hard disks. We equip 42 of them. While fourty disks are used as storage to be exported two disks are used for the operating system. These two disks can be somewhat smaller.

In the follwing calculations we will evaluate four different scenarios:

• 40 disks with 1 TB capacity each
• 40 “server” disks with 1 TB capacity each
• 40 disks with 1.5 TB each
• 40 disks with  2 TB each

For the operating system we choose two disks with 500 GB each. The total price of these two disk is approx. 100 Euros or 148 US-$.

Scenario #1: 40 disks with 1 TB each

• 1 TB HDD: 53 Euro (for instance: “Maxtor DiamondMax 23″)
• 40x 1 TB HDDs: 2120 Euros

Scenario #2: 40 “server” disks with 1 TB capacity each

• 1 TB “server” HDD: 84 Euros (for instance: “Samsung F1 RAID Class”)
• 40x 1 TB “server” HDDs: 3360 €

Scenario #3: 40 disks with 1.5 TB each

• 1.5 TB HDD: 84 Euros (for instance: “Seagate Barracuda 7200.11″)
• 40x 1.5 TB HDDs: 3360 Euros

Scenario #4: 40 disks with 2 TB each

• 2 TB HDD: 134 Euro (for instance: “WD20000CSRTL2″)
• 40x 2 TB HDDs: 5360 Euros

Calculations

Useable Capacity

• Useable capacity with 1 TB disks:
  • 2 pools x (20-1 Hot Spare – 2 Parity ) x 1000 GB ~ 34 TB
  • usable capacity approx. 31.6 TB
• Useable capacity with 1,5 TB Disks:
  • 2 Pools x (20-1 HS – 2 Parity ) x 1500 GB ~ 51 TB
  • usable capacity approx. 47.5 TB
• Useable capacity with 2 TB Disks:
  • 2 Pools x (20-1 HS – 2 Parity ) x 2000 GB ~ 68 TB
  • usable capacity approx. 63.3 TB

Throughput calculation

• 34 data disks total
• tecnical data details for Seagate 1,5 TB disk:
  • Spindle Speed      7,200 rpm
  • Average latency     4.16 msec
  • Random read seek time      <8.5 msec
  • Random write seek time     <10.0 msec
  • Calculation: → avg. seek time = (8,5+10)/2 = 9,25 ms (for a 50/50 mixture of reads and writes)
• IO calculations:
  • Rotation per milli secons = 7200 rpm / 60000 = 0,12 ms
  • Full Rotation Time = 1/rot. per ms = 1/0,12 = 8.33 ms
  • avg. rot. latency = 8,33 / 2 = 4,17 ms
  • IO time in ms = avg. seek time + rot. Latency = 4,17 + 9,25 = 13.42 ms
  • IOPS = (1/IO time*1000) = 74.53 IOPS
• Total possible IOPS: 34 Disks a 74 IOPS ~~ 2516 IOPS

Costs calculation

System components

Harddisks

Usage scenarios

• cheap, but reliable and performant multi-purpose storage
• if one system seems not reliable enough:
  • use two of them
  • and mirror mith oracle ASM for highest availability

The authors

Name	Volodymr Dubinin	Ronny Egner
Company	SIV.AG Konrad-Zuse-Straße 1 18184 Rogentin Germany	Ronny Egner Consulting Vinckestraße 22 40470 Dusseldorf Germany
EMail	vd@siv.de	ronnyegner@gmx.de