RMAN-00571: ===========================================================
RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============
RMAN-00571: ===========================================================
ORA-07217: sltln: environment variable cannot be evaluated.

It turned out the script used to start the rman backup was the problem:

connect target /
connect catalog rman/rman@catalog
run {
backup database;
}

After changing the target connection line to include the username, password and TNS connect string the backup ran fine:

connect target sys/sys@database
connect catalog rman/rman@catalog
run {
backup database;
}

nsraddadmin -u "user=administrator, host=machine1"

Just a short how to rename grid disks (and keep the size). In the example below the grid disks name contained a typo ‘FALSH02′ and had to be renamed to ‘FLASH02′:

dcli -c exadata01cel01 -l root "cellcli -e drop griddisk ALL prefix=FALSH02 force"
dcli -c exadata01cel01 -l root "cellcli -e creategriddisk ALL HARDDISK prefix=FLASH02"

Sample output:

[root@exadata01db01 ~]# dcli -c exadata01cel01 -l root "cellcli
-e drop griddisk ALL prefix=FALSH02 force"

exadata01cel01: GridDisk FALSH02_CD_00_exadata01cel01 successfully dropped
exadata01cel01: GridDisk FALSH02_CD_01_exadata01cel01 successfully dropped
exadata01cel01: GridDisk FALSH02_CD_02_exadata01cel01 successfully dropped
exadata01cel01: GridDisk FALSH02_CD_03_exadata01cel01 successfully dropped
exadata01cel01: GridDisk FALSH02_CD_04_exadata01cel01 successfully dropped
exadata01cel01: GridDisk FALSH02_CD_05_exadata01cel01 successfully dropped
exadata01cel01: GridDisk FALSH02_CD_06_exadata01cel01 successfully dropped
exadata01cel01: GridDisk FALSH02_CD_07_exadata01cel01 successfully dropped
exadata01cel01: GridDisk FALSH02_CD_08_exadata01cel01 successfully dropped
exadata01cel01: GridDisk FALSH02_CD_09_exadata01cel01 successfully dropped
exadata01cel01: GridDisk FALSH02_CD_10_exadata01cel01 successfully dropped
exadata01cel01: GridDisk FALSH02_CD_11_exadata01cel01 successfully dropped

[root@exadata01db01 ~]# dcli -c exadata01cel01 -l root "cellcli
-e create griddisk ALL HARDDISK prefix=FLASH02"

exadata01cel01: GridDisk FLASH02_CD_00_exadata01cel01 successfully created
exadata01cel01: GridDisk FLASH02_CD_01_exadata01cel01 successfully created
exadata01cel01: GridDisk FLASH02_CD_02_exadata01cel01 successfully created
exadata01cel01: GridDisk FLASH02_CD_03_exadata01cel01 successfully created
exadata01cel01: GridDisk FLASH02_CD_04_exadata01cel01 successfully created
exadata01cel01: GridDisk FLASH02_CD_05_exadata01cel01 successfully created
exadata01cel01: GridDisk FLASH02_CD_06_exadata01cel01 successfully created
exadata01cel01: GridDisk FLASH02_CD_07_exadata01cel01 successfully created
exadata01cel01: GridDisk FLASH02_CD_08_exadata01cel01 successfully created
exadata01cel01: GridDisk FLASH02_CD_09_exadata01cel01 successfully created
exadata01cel01: GridDisk FLASH02_CD_10_exadata01cel01 successfully created
exadata01cel01: GridDisk FLASH02_CD_11_exadata01cel01 successfully created"

set linesize 100
set pagesize 100

select
                a.tablespace_name,
                round(SUM(a.bytes)/(1024*1024*1024)) CURRENT_GB,
                round(SUM(decode(b.maxextend, null, A.BYTES/(1024*1024*1024),
                b.maxextend*8192/(1024*1024*1024)))) MAX_GB,
                (SUM(a.bytes)/(1024*1024*1024) - round(c.Free/1024/1024/1024)) USED_GB,
                round((SUM(decode(b.maxextend, null, A.BYTES/(1024*1024*1024),
                b.maxextend*8192/(1024*1024*1024))) - (SUM(a.bytes)/(1024*1024*1024) -
                round(c.Free/1024/1024/1024))),2) FREE_GB,
                round(100*(SUM(a.bytes)/(1024*1024*1024) -
                round(c.Free/1024/1024/1024))/(SUM(decode(b.maxextend, null, A.BYTES/(1024*1024*1024),
                b.maxextend*8192/(1024*1024*1024))))) USED_PCT
from
                dba_data_files a,
                sys.filext$ b,
                (SELECT
                               d.tablespace_name ,sum(nvl(c.bytes,0)) Free
                FROM
                               dba_tablespaces d,
                               DBA_FREE_SPACE c
                WHERE
                               d.tablespace_name = c.tablespace_name(+)
                               group by d.tablespace_name) c
WHERE
                a.file_id = b.file#(+)
                and a.tablespace_name = c.tablespace_name
GROUP BY a.tablespace_name, c.Free/1024
ORDER BY tablespace_name;

The result looks like this:

TABLESPACE_NAME                CURRENT_GB     MAX_GB    USED_GB    FREE_GB   USED_PCT
------------------------------ ---------- ---------- ---------- ---------- ----------
SYSAUX                                 16         32          1         31          3
SYSTEM                                 16         32          0         32          0
TEST                                   10         11          0      10.85          0
UNDOTBS1                               16      32768          0      32768          0
UNDOTBS2                               16      32768          0      32768          0
UNDOTBS3                               16      32768          0      32768          0
UNDOTBS4                               16      32768          0      32768          0
UNDOTBS5                               16      32768          0      32768          0
UNDOTBS6                               16      32768          0      32768          0
UNDOTBS7                               16      32768          0      32768          0
UNDOTBS8                               16      32768          0      32768          0
USERS                                   1         32          0         32          0

Before we start a preleminary note: It is a very good idea (i´d call it: “required”) to use separate subnets for each physical interface.  Do NOT use the same subnet accross different network interfaces!

If you do not comply with this simple rule you will end up having problems with so called Arp Flux (also documented here, here, here and so on) which requires further modifications.

For configuring and using iSCSI multipathing the following packages are needed:

• device-mapper-multipath
• device-mapper-multipath-libs
• iscsi-initiator-utils

Our testlab used a VM based on Oracle Enterprise Linux 6 Update 2 with two physical interfaces:

• eth1:    192.168.1.200/24
• eth3:    192.168.10.2/24

Multipathing

Why choosing Multipath over network bonding?

Multipathing and network bonding both protect against failing network devices such as cables, switches and network cards and at the same time provde higher throughput by using more than one interface.

Problem 1: Speed aggregation

When using network bonding for port aggregation you need to decide which algorithm should be used to distribute the packages over the available interfaces. For all except the ‘active / backup’ algorithm both the switch and the operating system must support the chosen algorithm. This includes the widely used LACP algorithm. Another downside is that algorithms such as LACP cannot be used accross physically independent switches.

In addition to that all algorithms except the ’round robin’ altgorithmn do NOT offer a speed improvement for a single connection beyond the speed of a single interface. However by distributing the different connections over all available interfaces the total throughout for all connections can indeed be higher than 1 Gbit/s (if your systems consists of several NICs with 1 Gbit/s speed).

The only exception to this rule is the ’round robin’ algorithm which offers the whole aggregated network speed even for a single connection. The downside is that most switches do not support this algorithmn.

In contrast Multipathing offers real aggregation by distributing the data over all available paths hence effectively aggregating the throughput.

Problem 2: High Availability

While every network bonding algorithmn protects against failing network cards and cables protection against failing switches is difficult. All algorithmns except ‘active / backup’ require the switches being used to communicate with each other and explicitly offer support for this configuration which is only available in expensive switches.

Multipathing handles each connection separately and does not require any support in the switches firmware.

Basic Configuration

Create a file /etc/multipath.conf with the following content:

[root@mhvtl media]# cat /etc/multipath.conf
 defaults {
 udev_dir                /dev
 polling_interval        10
 path_selector           "round-robin 0"
 path_grouping_policy    multibus
 path_checker            readsector0
 rr_min_io               100
 max_fds                 8192
 rr_weight               priorities
 failback                immediate
 no_path_retry           fail
 user_friendly_names     yes
 }

Note that

     no_path_retry    fail

is required for any kind of multipathing to work. Without this setting I/Os on failed paths end up being queued causing the whole device to become unresponsible.

iSCSI: Connecting to the Target

Step #1: Discover iSCSI Targets

[root@mhvtl iscsi]# iscsiadm -m discovery -t sendtargets -p  192.168.10.1
 Starting iscsid:                                           [  OK  ]
 192.168.10.1:3260,2 iqn.1986-03.com.sun:02:f1bb8f6d-b6d2-c3f7-cfb9-ff56689144d2
 192.168.1.5:3260,3 iqn.1986-03.com.sun:02:f1bb8f6d-b6d2-c3f7-cfb9-ff56689144d2

Step #2: Login to the Targets

Portal #1: 192.168.10.1

[root@mhvtl iscsi]# iscsiadm -m node --target iqn.1986-03.com.sun:02:f1bb8f6d-b6d2-c3f7-cfb9-ff56689144d2
--portal 192.168.10.1 --login

Logging in to [iface: default, target: iqn.1986-03.com.sun:02:f1bb8f6d-b6d2-c3f7-cfb9-ff56689144d2,
portal: 192.168.10.1,3260] (multiple)

Login to [iface: default, target: iqn.1986-03.com.sun:02:f1bb8f6d-b6d2-c3f7-cfb9-ff56689144d2,
portal: 192.168.10.1,3260] successful.

Portal #2: 192.168.1.5

[root@mhvtl iscsi]# iscsiadm -m node --target iqn.1986-03.com.sun:02:f1bb8f6d-b6d2-c3f7-cfb9-ff56689144d2
--portal 192.168.1.5 --login

Logging in to [iface: default, target: iqn.1986-03.com.sun:02:f1bb8f6d-b6d2-c3f7-cfb9-ff56689144d2,
portal: 192.168.1.5,3260] (multiple)

Login to [iface: default, target: iqn.1986-03.com.sun:02:f1bb8f6d-b6d2-c3f7-cfb9-ff56689144d2,
portal: 192.168.1.5,3260] successful.

Status query

[root@mhvtl iscsi]# iscsiadm --mode node
 192.168.1.5:3260,3 iqn.1986-03.com.sun:02:f1bb8f6d-b6d2-c3f7-cfb9-ff56689144d2
 192.168.10.1:3260,2 iqn.1986-03.com.sun:02:f1bb8f6d-b6d2-c3f7-cfb9-ff56689144d2

/var/log/messages

Connection to portal 192.168.10.1:

 Jun 24 18:03:51 mhvtl kernel: scsi6 : iSCSI Initiator over TCP/IP
 Jun 24 18:03:51 mhvtl kernel: scsi 6:0:0:0: Direct-Access     NEXENTA  NEXENTASTOR      1.0  PQ: 0 ANSI: 5
 Jun 24 18:03:51 mhvtl kernel: sd 6:0:0:0: Attached scsi generic sg3 type 0
 Jun 24 18:03:51 mhvtl kernel: sd 6:0:0:0: [sdc] 10485760 512-byte logical blocks: (5.36 GB/5.00 GiB)
 Jun 24 18:03:51 mhvtl kernel: sd 6:0:0:0: [sdc] Write Protect is off
 Jun 24 18:03:51 mhvtl kernel: sd 6:0:0:0: [sdc] Write cache: enabled, read cache: enabled,
 doesn't support DPO or FUA
 Jun 24 18:03:51 mhvtl kernel: sdc: detected capacity change from 0 to 5368709120
 Jun 24 18:03:51 mhvtl kernel: sdc: sdc1
 Jun 24 18:03:51 mhvtl kernel: sd 6:0:0:0: [sdc] Attached SCSI disk
 Jun 24 18:03:51 mhvtl multipathd: sdc: add path (uevent)
 Jun 24 18:03:51 mhvtl kernel: device-mapper: multipath round-robin: version 1.0.0 loaded
 Jun 24 18:03:51 mhvtl multipathd: mpatha: load table [0 10485760 multipath 0 0 1 1 round-robin 0 1 1 8:32 1]
 Jun 24 18:03:51 mhvtl multipathd: mpatha: event checker started
 Jun 24 18:03:51 mhvtl multipathd: sdc path added to devmap mpatha
 Jun 24 18:03:52 mhvtl iscsid: Connection4:0 to [target: iqn.1986-03.com.sun:02:f1bb8f6d-b6d2-c3f7-cfb9-
 ff56689144d2, portal: 192.168.10.1,3260] through [iface: default] is operational now

Connection to portal 192.168.1.5:

 Jun 24 18:04:01 mhvtl kernel: connection3:0: detected conn error (1020)
 Jun 24 18:04:01 mhvtl iscsid: Connection3:0 to [target: iqn.1986-03.com.sun:02:f1bb8f6d-b6d2-c3f7-cfb9-
 ff56689144d2, portal: 192.168.1.5,3260] through [iface: default] is shutdown.
 Jun 24 18:04:03 mhvtl kernel: scsi7 : iSCSI Initiator over TCP/IP
 Jun 24 18:04:04 mhvtl kernel: scsi 7:0:0:0: Direct-Access     NEXENTA  NEXENTASTOR      1.0  PQ: 0 ANSI: 5
 Jun 24 18:04:04 mhvtl kernel: sd 7:0:0:0: Attached scsi generic sg4 type 0
 Jun 24 18:04:04 mhvtl kernel: sd 7:0:0:0: [sdd] 10485760 512-byte logical blocks: (5.36 GB/5.00 GiB)
 Jun 24 18:04:04 mhvtl kernel: sd 7:0:0:0: [sdd] Write Protect is off
 Jun 24 18:04:04 mhvtl kernel: sd 7:0:0:0: [sdd] Write cache: enabled, read cache: enabled,
 doesn't support DPO or FUA
 Jun 24 18:04:04 mhvtl kernel: sdd: detected capacity change from 0 to 5368709120
 Jun 24 18:04:04 mhvtl kernel: sdd: sdd1
 Jun 24 18:04:04 mhvtl kernel: sd 7:0:0:0: [sdd] Attached SCSI disk
 Jun 24 18:04:04 mhvtl multipathd: sdd: add path (uevent)
 Jun 24 18:04:04 mhvtl multipathd: mpatha: load table [0 10485760 multipath 0 0 1 1 round-robin 0 2 1 8:32 1 8:48 1]
 Jun 24 18:04:04 mhvtl multipathd: sdd path added to devmap mpatha
 Jun 24 18:04:04 mhvtl iscsid: Connection5:0 to [target: iqn.1986-03.com.sun:02:f1bb8f6d-b6d2-c3f7-cfb9-
 ff56689144d2, portal: 192.168.1.5,3260] through [iface: default] is operational now

Multipath Status

[root@mhvtl iscsi]# multipath -ll
 mpatha (3600144f02ba6460000004fe7032f0001) dm-1 NEXENTA,NEXENTASTOR
 size=5.0G features='0' hwhandler='0' wp=rw
 `-+- policy='round-robin 0' prio=1 status=active
 |- 6:0:0:0 sdc 8:32 active ready running
 `- 7:0:0:0 sdd 8:48 active ready running

Testing Multipath

After setting up and configuring multipathing it is highly recommended to test the behavior if paths fail. To do a simple test i stopped the interface “eth3″ which leaves one remaining interface:

ifconfig eth3 down

/var/log/messages shows:

 Jun 24 18:05:34 mhvtl kernel: connection4:0: ping timeout of 5 secs expired, recv
 timeout 5, last rx 4299040130, last ping 4299045130, now 4299050130
 Jun 24 18:05:34 mhvtl kernel: connection4:0: detected conn error (1011)
 Jun 24 18:05:35 mhvtl multipathd: mpatha: sdc - readsector0 checker reports path is down
 Jun 24 18:05:35 mhvtl multipathd: checker failed path 8:32 in map mpatha
 Jun 24 18:05:35 mhvtl multipathd: mpatha: remaining active paths: 1
 Jun 24 18:05:35 mhvtl kernel: device-mapper: multipath: Failing path 8:32.
 Jun 24 18:05:35 mhvtl iscsid: Kernel reported iSCSI connection 4:0 error (1011) state (3)

As you can see multipathd recognized the failing path and stopped I/O using the underlying device (in our test “/dev/sdc”):

[root@mhvtl media]# multipath -ll
 mpatha (3600144f02ba6460000004fe7032f0001) dm-1 NEXENTA,NEXENTASTOR
 size=5.0G features='0' hwhandler='0' wp=rw
 `-+- policy='round-robin 0' prio=1 status=active
 |- 6:0:0:0 sdc 8:32 failed faulty running
 `- 7:0:0:0 sdd 8:48 active ready  running

In our test there was a small I/O stall of approx 5 seconds after which everything continued to function normally. When re-enabling network interface eth3 multipathd automatically repaired the faulty path.

In my lab setup i had a machine with two physically separated interfaces:

• e1000g0: 192.168.1.5/24   and
• e1000g1: 192.168.10.1/24

As you can see both interfaces are on different subnets. Note that it is a very good idea (i´d call it: “required”) to use a separate subnet for each interface.  It will save you a lot of trouble and problems!

If you do not comply with this simple rule you will end up having problems with so called Arp Flux (also documented here, here, here and so on) which requires further modifications.

Note: If you modify and existing installation you only need to execute Step #2 and Step #3. Everything else should already be there.

Step #1 : Create an iSCSI target

This step is only required if you do not yet have one. If you already have one – skip this step.

Enter a desired target name into the name field or leave it blank to automatically create one:

The created target looks like this:

Step #2: Create Target Portal Groups

You need to create one Target Portal Group for each interface you want to use for iSCSI. In our case we will create two Portal Groups because we want to use two interfaces:

The result looks like this:

Step #3: Reconfigure iSCSI Target

In the next step we add our created target portal groups to the iSCSI target (created in Step #1):

The result:

Bascially were done here. Step 4 – 8 shows how to create a LUN and map these LUN to target and initiator.

Step #4: Create Target Group

If you do not already have a target group create one and add your iSCSI target. If you already have a target group skip this step.

Step #5: Create a ZVOL

A ZVOL is the LUN which will be visible on the initiator:

Step #6: Create Remote Initiator

To map a LUN (or ZVOL) to the appropriate clients you need to create a Initiator Group. These group consists of remote initiator names. The initiator name can either be found on the client e.g. Linux stores the initiatorname in /etc/iscsi/initiatorname.iscsi) or – if the client is already connected to the target – it will be display in Step 7.

If you want or need to create the remote initator manually you can do this under ‘Remote Initiator’:

Step #7: Create Remote Initiator Group

This step finally creates the remore initiator group and assigns a name to them:

Step #8: Map Target Group, Initiator Group and ZVOL

The last step finally maps the ZVOL (= the LUN to be exported), the initiator group (= the client(s) you want to export the LUn to) and the target group (the protocol you want to use to export the LUN):

In addition to the kernel modules you need the following packages:

• oracleasm-support
• oracleasmlib

For Oracle Enterprise Linux “oracleasm-support” can be found on the installation media while “oracleasmlib” is not available on the installation media. “oracleasmlib” can be downloaded over the ULN network (more information here).

If you dont have a ULN account or using Red Hat you can get the freely available packages (for OEL and Red Hat) here.

Normally changing the driver for the syspool requires a complete reinstallation. However if you want to avoid this simple follow this guide:

1. Boot with Nexenta Setup CD; Choose “Rescue System”
2. when booted import the syspool with:

   zpool import -f syspool

3. Install boot loader on ALL disks belonging to the syspool, e.g. for the first disk:

   grub-install /dev/rdsk/c0t0d0s0

4. Export the syspool:

   zpool export syspool

5. Reboot

Done. The system now uses AHCI drivers.

kernel: Exiting Time2Retain handler because session_reinstatement=1

This happens if you access LUNs from more multiple nodes (as required for RAC setups)  while all nodes share the same initiatorname.

The fix is to use different and unique initiatornames for every node.