Current hardware is powerful enough to allow “supernodes” that can run hundreds of databases. We are also confronted with Virtualization or “carve-up” of hardware by way of XEN, VMware, or vendor specific product that create domains or partitions.
With the current push towards “virtualization” of systems and the (in)capabilities of Oracle, it may just be worth to re-start some of the n:m discussion (did it ever go away?).

It is time to take a position on the many-to-many relationship between databases and systems.

To summarize it for those architects and (account-)managers who are in a hurry:
I am in favor of 1:1 wherever possible, I support 1:m (RAC, grid) if really, really needed (but just please think about it one more time). RAC is a wonderful piece of technology that can serve many other vendors as a good example. It can be made to work. I will try to indicate under which conditions I think RAC can or cannot be applied.
Finally, I will only tolerate the old-fashioned m:1 for non-critical situations or on systems where there is some sort of risk-mitigation against interference between the (instances of) multiple databases.

After explaining these positions, I also have a list of recommendations for customers, providers and even for the Big-Oracle itself.

Those of you interested: read on. Possibly prove me wrong.
Others, keep browsing, the truth is out there, whichever version or vendor you want to see.

For the record: the bottom of the text contains links to all our major “partners”. Please keep the invites coming.

Intro, and some definitions.

On conventional systems, we generally find 1 or more databases running on a single system (*nix or even windows). For example, for concept-testing of a DG setup with cascaded standby, my laptop has run 5 databases simultaneously. Slow, but running. It just takes loads of memory and careful (memory-) parameter setting. It illustrates the capabilities of the Oracle database that the concept, once proven on a laptop could then be used to clone a 3TB production system (this time not on Windows). But the laptop Proof-of-Concept also illustrated an important issue when running multiple databases on a single system: Contention. The databases were visibly (and audibly from disk and fan) competing for IO and CPU resource.

NB: Can’t resist to put one in for Deb and Nigel: Let the world know that Deb got more memory into her laptop then Nigel, but then both David and I still have more memory in our laptops then the two of you together, so there ;-). Size matters. Techies rule.
Back to serious business.

With the current possibilities for virtualization, you can take hardware and split it into many “systems” using VMware, Xen or vendor-specific tools for domains, lpars, etc.. Each resulting system can then be used to run instance(s) of 1 or more databases. When is this useful and how far should we take this?
With Oracle RAC (GRID, anyone?) you can take a database and distribute its instances over many system. When does this add benefits?

First a few quick definitions to delimit the playing field. Please not that the definitions are for the purpose of this argument only. They don’t pretend to be scientific or final. Or even to be correct.

System: a running *nix (or win*) instance containing a process list and an amount of addressable memory. On short: a running instance of an “operating system”.
A system can be Virtual if the hardware seen by the operating system is not identical to the actual underlying kit. This is the case when a larger system is split into “virtual” units by use of Xen, VMware or some vendor-specific layer of software or firmware. Some of those can also be modified dynamically (e.g.: Rolling). Some definitions and a good description can be found here:
http://encyclopedia.thefreedictionary.com/virtualization.

Database: an (Oracle) database, containing one system tablespace, and one user called SYS (I’m still trying to find the “essence” of an Oracle database, how about the sys.obj$ entries?). Note that the use of DataGuard can mystify the definition of a “database”, because each DG-clone can represent “the database”. The actual Single Point of Truth (SPoT) is where the current “primary instantiation” of the database resides.

Also note that my definitions do not include the binaries, or the ORACLE_HOME, as part of the database or the system. Indeed, systems and databases can be used in situations where the software needed to run them is “shared”. Most system only need a few “system specific” files in /var and /etc. I will always point out VMS as the ultimate mother of all clustered systems whereby files are shared between multiple nodes. But that requires a Clustered File System (CFS), and that opens a different discussion altogether. Suffice to say that a CFS is very suitable to ensure that all machines can be connected to the same, identical software and are guaranteed to run the same version of the binaries.

Now let me briefly elaborate on the different options.

Conventional: many:1.

On conventional systems, we often see many databases running on a single (unix) instace.
The DBA can look after the databases, and the unix administrators have only one entity to watch. Any collision between databases will have to be handled by the DBA. Note that in most of these conventional cases all databases share the same software-tree (multi databases running from the same oracle_home).
These systems tend to have a relaxed SLA. The utilization varies and sometimes we see high percentages of CPU or IO bandwidth being consumed by a single database.

The main disadvantage of conventional systems is the fact that there is only one system, and all databases meet there. A problem with either a single database or with the system itself can quickly contaminate all databases on the system. And upgrades of system- or oracle-software lead to simultaneous outages of all databases on the system.

In case of system- or hardware failures many databases must be recovered simultaneously or re-started on one or more (other) systems (requiring some prioritization). The simultaneous recovery of many databases may lead to a brief period of overload and/or chaos an possibly a domino effect on other systems or components.

The advantages of an m:1 configuration are the simplicity of a “single system”, which is easy on the system-admin, and the often cheaper license structure when using per-system license.

Simple: 1:1

On system with a high load or a stringent SLA, we tend to see 1:1 relationships: a single database, whereby the sole and single instance of the database runs in a single unix system. This is sometimes referred to as the monogamous configuration.
By having the whole system to itself, the database can benefit from all the resources available. There is no interaction (disturbance) from adjacent instances or other processes. Determining parameters is relatively simple.

However, as Oracle (-sales) will point out, this 1:1 configuration generally means the system is grossly under-utilized. It also means the database can still suffer from unix- or hardware failures.
But I like the simplicity of this configuration, I think this is the most “robust” solution and is applicable to the majority of databases. Whenever a problem occurs on one of the systems, only one database is affected, and recovery-efforts can be concentrated on a single database and application, reducing the risk of a domino-effect.

The 1:1 situation also lends itself very well to hardware-clustering or “cold failover” whereby a database is re-started on another system (node) in case the underlying system or hardware fails. Only one database needs to be re-started or recovered.

Since a 1:1 configuration requires many “systems”, it is attractive to use server virtualization. By running multiple “virtual” systems on a single piece of hardware, you can quickly create the required number of “separate” or isolated systems. When doing this, keep in mind that the underlying hardware remains the single point of failure. When one or more virtual system are meant to replace one-another in case of failure, they should preferably run on separated hardware.

RAC: 1 : many.

As a techie, I like the technology behind RAC. It is a wonderful thing to play with and I like the challenge to master this thoroughbred in real-life system. But I have to be careful not to be running a “solution looking for a problem”.

We tend to see RAC databases in organizations with formal and very stringent SLA’s and with the budget and the resources to try and meet these requirements.

OPS and RAC eliminate the SPoF of the “system” and deploy the database over multiple systems. Theoretically this works nicely and even provides dynamic provisioning of system resources (you can utilize all available kit, and you can add more kit as needed). In practice, many have pointed out the relative complex setup, the high price, and the other shortcomings of RAC (link to Miraculous, Famous Danish company).

We have indeed seen successful deployments of multi-instance database on some very large kit. In some cases, the impressive amount of hardware was able to hide badly designed application-code for quite some time. And by constantly distributing the (mainly CPU-) workload over the available unix systems, the coders got away with some appallingly inefficient constructions. Some of these cases have demonstrated the viability (and sometimes vulnerability) of RAC quite nicely, although a better design or implementation might have been cheaper (I prefer brains over iron, always!, in case of doubt: reduce the size of the hardware and tell the IT crowd to JFDI).
Note however, that it is important to let the hardware boys and vendors it their way a bit too. Riding in some extra hardware makes them happy, and is good for our relationship with these vendors. They might invite us on future projects.

And to pour further praise on the Oracle techies, TAF has saved our systems several times when a node got in trouble and died. Database-nodes die mostly through software errors, core dumps or memory leaks, and sometimes through human errors. The underlying hardware is rarely a problem, as these high-end-hardware systems are built to keep running run even if a salty ocean wave runs through the lower floors of the building : unintentionally (Kathryn, are you reading this?).
Please remark that even when you have RAC and TAF-capabilities, you still need to code your application to correctly trap and handle the failover-events. Otherwise, it only works on “idle” connections.

Note that most databases can (be made to) run on RAC, provided there are no obvious bottlenecks such as an ordered-no-cache sequence or some very hot blocks with running-totals. And those bottlenecks can generally be un-designed.
NB: Oracle currently seems to have the following position on RAC: if it doesn’t run (or scale) on RAC, your design is wrong.

Suffice to say that where Very Fast Failover (TAF or FCF/FAN) are needed, RAC has no equal. And for systems that have extreme hardware (CPU) requirements, the RAC scale-out model is also beneficial.

Question : Now what to choose ?

The classic answer: It Depends.
However, I will try to provide some guidance and some opinions.

The only factor that really matters is “The Business” (duh).
What does your business need and what can it afford. For simple or undemanding SLAs the traditional m:1 configuration is often sufficient and cost-effective. For businesses that have more stringent demands or for providers that risk being sued by their (business-)customers over a broken SLA, a 1:1 is advised, possibly with some cold-failover mechanism. And finally, if you really need the additional 10 minutes, or if you need the scale-out features, AND if you can afford the resources for testing, training and ongoing maintenance, a 1:m configuration (aka RAC or grid) can be your choice.

To figure out what your business needs (and can afford), you can either think for yourself, or you can give yourself and your department more credibility by engaging (principle-, business-) consultants to do cost-benefit analysis, risk-assessments or FEMA (that is: Failure Escalation Mode Analysis, not the other FEMA). They will especially stress the business-cost of downtime and any related loss of data/productivity/customers/orders/MegaWatts/ That is enough FUD for the moment. Back to more practical matters.

On the practical level, there are some factors that come into play. There is a) the preference (eeh: dictate) of your system administrators, your SAN engineers, your ASP, or your hosting provider. Then there is b) your commercial relationship with Oracle which will determine how high your licensing cost will turn out. But there may be others, such as c) the capabilities and preferences of your DBA. We will not even go into items like d) the availability of test-systems to prove and maintain your architecture, or e) the available rack-space.

The first important factor that often comes into play is the preference (or the pricing-policy) of the system-admin team or the Hosting-Organization (the ASP). Are they capable of handling many systems or do they prefer a low number of unixes ? Can they quickly build and clone systems for provisioning? What price do your ASP’s charge for additional systems ? This may determine your capability to run “multiple” systems.
Some organizations, by choice or by force, still get away with running just 1 large unix box with everything on it: HR, CRM, Logistics, and sometimes they even have their dev/test/uat environments on the very same box. Feasible, but with most of the drawback of a m:1 configuration.

The next factor is often License cost. How is your relation with Oracle, commercially? If you have to pay list-price, you will want to stick with “conventional”. Here, Oracle shoots itself in the foot: a lower price on RAC would speed up acceptance of the RAC and GRID model.
Machiavelli did suspect this was done to buy sufficient Beta-time to find all quirks, possibly to find a solution for instance-affinity and to give customers the time to come up with a solution for the friendly delivered “Your Design is Wrong” consultancy-audit outcomes.

And last but not least, what do your DBA’s prefer, and how trained and comfortable are they with RAC/grid? The traditional choice, m:1, despite its disadvantages (contention, domino-effects) is still the easiest to maintain for a DBA. Choosing a 1:1 configuration brings on a slightly higher workload, but has the advantage of more robustness and easier, isolated, troubleshooting since databases and systems do not affect one-another when trouble or maintenance occurs (yes, yes, someone must shout “utilization” now, thank you).

The choice for a RAC or grid configuration tends to create a significant overhead. We politely disagree with Oracle at this point that the new grid-control alleviates all problems.
And even if GC and its agents do try to take away a lot of the routine-tasks, Knowledge and Experience can never be completely replaced by a GUI. This aspect, the Human- or Operator dimension, tends to be the most under-estimated factor when (prematurely) implementing RAC/Grid.

Recommendations:

For customers and end-user businesses:
Move carefully from m:1 to 1:1. The 1:1 configuraion is at this moment arguably the most robust way to run a database. Consider using virtual systems to support a 1:1 deployment, but beware of the possible contention and SPoF on the underlying physical layers. Move on to 1:m (RAC) for cases with specific needs (failover or scale-out). Only use RAC in cases where you must, but then don’t hesitate to use it. It can (be made to) work, and it will work (eventually), and you may have to learn these tricks eventually. Start on the first valid occasion.

For ASP and hosting organizations:
Learn how to handle clusters, clones and virtual systems. These tools will give you an edge in flexibility. Then offer your customers the possibility to host many _identical_ systems at a rebate. Your customers will buy more as 1:1 and 1:m systems proliferate, and in the long run you will benefit. Hosting Companies and some vendor pricing-policies are the largest obstacle when moving from m:1 to 1:1 or even on to 1:m. Innovative customers will try to move to cheaper and more flexible platforms, and even lagging customers will eventually follow. If the hosting provider can quickly “provision” at acceptable cost, he can be seen as a partner in commoditization, rather then as an obstacle to flexibility.

For System-admins:
Learn how to handle a multitude of systems, learn how to keep them in sync, and how to clone or (re-)build systems quickly.
NB: for the addicts: investigate the use of Clustered File Systems (CFS).

For DBA’s and system-admins:
Aim to deploy databases and systems in a 1:1 fashion. The “isolation” of each database and system greatly facilitates admin- and troubleshooting activities.
Also get used to replicating or sharing software through OUI or other mechanisms.
And if possible, start to work with a CFS.
Sharing storage at the “filesystem” level can facilitate the juggling of multiple systems very much. Even NFS (supported, but not recommended is a usable alternative.
A CFS can offer great advantages by sharing files across nodes and this can simplify software-deployment and distribution. You will always need two copies of software for redundancy-purposes, but please think before making he 3rd, 4th or 42nd copy. Sharing is better the copying, especially at high numbers. It is easier to manage a small number of shared oracle_home trees then to have 42 or more copies that need to be rsynced or otherwise kept identical.

For Oracle:
It is appalling to find that a grid (aka an OPS database) was easier to build and maintain under VMS or Tru64 with Oracle 8174 then it is with the current 10g versions.
Please pursue the development of OCFS and facilitate shared-binary installs on OCFS and other CFS platforms. This will help proliferation of your GRID strategy, and will get you more market-share and revenue in the long run. The oracle-inventory mechanism and the configuration of agents tend to make life difficult for deployment of shared-binaries. Any viable grid should IMHO include the shared use of a software tree and not depend on endless replication of executables.

Shutdown (normal)

By adopting a grid-strategy, Oracle has greatly increased the options for its customers. And for those of you who don’t know it yet: find the Oracle Sponsored GRID-INDEX. Various hard- and software vendors have added to the palette of choices by implementing their own versions of Grid, clusters, or virtualization. All this new (eh, apologies: innovative but proven) technology can be put to good use, but only when making the correct choices.

We hope the preceding information can offer some help, and we would like to close down by adding one more item of advice: try to base you choices on simplicity

Relevant links

The usual brownie-point partner-links and marketing buzzwords on grid etc :
http://www.oracle.com/global/nl/corporate/press/200634.html
http://www.oracle.com/global/eu/pressroom/emeagridreport4.pdf
http://www.hp.com/techservers/grid/index.html
http://www.ibm.com
http://www.sun.com

More on server-virtualisation (start here!):
http://encyclopedia.thefreedictionary.com/virtualization

An introduction to Clustered File Systems:
http://www.beedub.com/clusterfs.html
http://www.oracle.com/technology/pub/articles/calish_filesys2.html

Some good, albeit biassed arguments for CFS can also be found here:
http://www.polyserver.com (look for articles by Kevin Closson).

A classic on RAC:
http://www.miracleas.dk/WritingsFromMogens/YouProbablyDontNeedRACUSVersion.pdf

Just for fun:
http://www.userfriendly.org
http://www.dilbert.com

Last bootnote for all Alliance Managers and other people in control of party-invites: take note of my cross-links to our “powerful partners”. And don’t worry, since you didn’t bother to read all of the text, neither did the real decision-makers, hence no damage is done. Oh, and please keep me on the invite-lists, I still appreciate good food and quality entertainment.

Now shoot me (over lunch?).

When other instances are active, an attempt to switch over fails with the following error message:

SQL> ALTER DATABASE COMMIT TO SWITCHOVER TO STANDBY;

ALTER DATABASE COMMIT TO SWITCHOVER TO STANDBY *

ORA-01105: mount is incompatible with mounts by other instances

Switch over steps

On the primary database, execute the following statement:

ALTER DATABASE COMMIT TO SWITCHOVER TO PHYSICAL STANDBY WITH SESSION SHUTDOWN;

This statement does the following:

- It closes the primary database, terminating any active sessions

- It transmits any unarchived redo log files and applies them to the standby database

- It adds an end-of-redo marker to the header of the last log file being archived

- It creates a backup of the current control file

- It converts the current control file into a standby control file

If the Primary Site is destroyed by a disaster you can bring up the standby database as the primary database on the standby database server with:

ALTER DATABASE RECOVER MANAGED STANDBY DATABASE CANCEL;

ALTER DATABASE ACTIVATE PHYSICAL STANDBY DATABASE;

The ACTIVATE statement immediately transitions the standby database to the primary role and makes no attempt to apply any additional redo data that might exist at the standby location. When using this statement, you must carefully balance the cost of data loss at the standby location against the potentially extended period of downtime required to fully repair the primary database.

Check the changed database role with:

select database_role, db_unique_name, name from v$database;

DATABASE_ROLE    DB_UNIQUE_NAME                 NAME

—————- —————————— ———

PRIMARY          stdb                           PRAC

Establishing a Time Lag on a Physical Standby Database

To create a physical standby database with a time lag, modify the

LOG_ARCHIVE_DEST_2 initialization parameter on the primary database to set a delay for the standby database. The following is an example of how to add a 4-hour delay:

alter system set log_archive_dest_2='SERVICE=stdb LGWR ASYNC

VALID_FOR=(ONLINE_LOGFILES, PRIMARY_ROLE) DB_UNIQUE_NAME=stdb' DELAY=240' scope=both;

See our PoC MAA- Blogs 5,6 and 7 about OCFS2, CRS and RAC install respectively.

Overview of the file system on linux5:

[oracle@linux5 ~]$ df -k

Filesystem           1K-blocks      Used Available Use% Mounted on

/dev/mapper/VolGroup00-LogVol00

                      67829336  11887112  52496672  19% /

/dev/cciss/c0d0p1       102182      6774     95408   7% /boot/efi

none                   1020720         0   1020720   0% /dev/shm

/dev/sdj                522240    105408    416832  21% /u01/cd1

/dev/sdk                522240    102336    419904  20% /u02/cd2

/dev/sdl                522240     86976    435264  17% /u03/cd3

/dev/sdm              20971200   3783872  17187328  19% /u01/sw1

/dev/sdn              20971200    137984  20833216   1% /u09/sw2

/dev/sdo              26213888  11774080  14439808  45% /u02/oradata

/dev/sdp              26213888   2737664  23476224  11% /u03/oraflra

We followed the steps outlined in Chapter 3 of the Oracle Data Guard Concepts and Administration Guide to create our physical standby database.

<!–More–>

 Step 1 Configure a Standby redo log

Ensure log file sizes are identical on the primary and standby databases:

The primary database has 8 redolog groups of 50Mb each (All 4 nodes have two groups each).

Because we use a backup to setup the Standby database the redologfiles of the Standby database are defacto the same size.

 Determine the appropriate number of standby redo log file groups:

As best practices you have to add:

(maximum number of logfiles for each thread + 1 ) * maximum number of threads = (2+1) * 4 = 12

 Verify related database parameters and settings:

Check MAXLOGFILES and MAXLOGMEMBERS settings for 10.1. databases.

Use: alter database backup controlfile to trace; + check the trace in the user_dump_dest directory. From 10.2 onwards the controlfile automatically expands for the MAX settings parameters. Be sure that 8 + 12 = 20 redolog files can be created.

If that is not the case re-create the controlfile on the primary database for 10.1. databases only.

Create standby redo log file groups:

Add 12 standby redolog groups on the primary database all of the same size with:

alter database add standby logfile group 9 ('/u02/oradata/prac/stredo09.log') size 50M;

alter database add standby logfile group 10 ('/u02/oradata/prac/stredo10.log') size 50M;

alter database add standby logfile group 11 ('/u02/oradata/prac/stredo11.log') size 50M;

alter database add standby logfile group 12 ('/u02/oradata/prac/stredo12.log') size 50M;

alter database add standby logfile group 13 ('/u02/oradata/prac/stredo13.log') size 50M;

alter database add standby logfile group 14 ('/u02/oradata/prac/stredo14.log') size 50M;

alter database add standby logfile group 15 ('/u02/oradata/prac/stredo15.log') size 50M;

alter database add standby logfile group 16 ('/u02/oradata/prac/stredo16.log') size 50M;

alter database add standby logfile group 17 ('/u02/oradata/prac/stredo17.log') size 50M;

alter database add standby logfile group 18 ('/u02/oradata/prac/stredo18.log') size 50M;

alter database add standby logfile group 19 ('/u02/oradata/prac/stredo19.log') size 50M;

alter database add standby logfile group 20 ('/u02/oradata/prac/stredo20.log') size 50M;

 Verify that the standby redo log file groups were created with:

select group#, thread#, sequence#, archived, status from v$standby_log;

    GROUP#    THREAD#  SEQUENCE# ARC STATUS

———- ———- ———- — ———-

         9   
       0          0 YES UNASSIGNED

        10          0          0 YES UNASSIGNED

        11          0          0 YES UNASSIGNED

        12          0          0 YES UNASSIGNED

        13          0          0 YES UNASSIGNED

        14          0          0 YES UNASSIGNED

        15          0          0 YES UNASSIGNED

        16          0          0 YES UNASSIGNED

        17          0          0 YES UNASSIGNED

        18          0          0 YES UNASSIGNED

        19          0          0 YES UNASSIGNED

        20          0          0 YES UNASSIGNED

12 rows selected.

Step 2 Set the Primary Database Initialization Parameters

Use the following alter system commands to change the necessary parameter settings:

alter system set log_archive_config='DG_CONFIG=(prac,stdb)' scope=both;

 alter system set log_archive_dest_1='LOCATION=/u02/oradata/prac_arch VALID_FOR=(ALL_LOGFILES, ALL_ROLES) DB_UNIQUE_NAME=prac' scope=both;

 alter system set log_archive_dest_2='SERVICE=stdb LGWR ASYNC VALID_FOR=(ONLINE_LOGFILES, PRIMARY_ROLE) DB_UNIQUE_NAME=stdb'  scope=both;

 alter system set log_archive_max_processes=30 scope=both;

 alter system set fal_server = 'stdb' scope=both;

 alter system set fal_client = 'prac' scope=both;

 alter system set db_file_name_convert='stdb','prac' scope=spfile;

 And the most important one: specify online redolog location on Primary and Standby location.                

alter system set log_file_name_convert = '/u02/oradata/prac/', '/u02/oradata/prac/' scope=spfile;

 Note that the scope is only spfile, so you have to restart the instance to make the parameter active.

 alter system set standby_file_management=AUTO scope=both;

Step 3 Check if ARCHIVELOG is enabled

Select log_mode from v$database;  Must be: ARCHIVELOG.

 If the database is in NOARCHIVELOG enable it by doing:

shutdown immediate;

startup mount;

alter database archivelog;

alter database open;

Step 4 Create a backup copy of the Primary Database datafiles

Use RMAN or even more simple use a COLD BACKUP.

 shutdown immediate;

 Copy all database files to the standby site

(This might take a while … depending on size of you’re database and network speed).

Step 5 Create a Control File and Parameter File for the Standby Database

startup mount;

alter database create standby controlfile as '/u02/oradata/prac/stdb.ctl';

create pfile='/u02/oradata/prac/initstdb.ora' from spfile;

alter database open;

 Modify this initstdb.ora file with “vi” according to the following:

This line is OK!! *.db_name='prac'

ADD line ==> *.db_unique_name='stdb'

 Flip names in these parameters to:

*.db_file_name_convert='prac','stdb'

*.log_archive_dest_1='LOCATION=/u02/oradata/prac_arch VALID_FOR=(ALL_LOGFILES, ALL_ROLES) DB_UNIQUE_NAME=stdb'

*.log_archive_dest_2='SERVICE=stby LGWR ASYNC VALID_FOR=(ONLINE_LOGFILES, PRIMARY_ROLE) DB_UNIQUE_NAME=prac'

*.fal_client='stdb'

*.fal_server='prac'

Remove both local and remote listener lines.

Save the file and
copy it with the standby controlfile to the standby database server.

 Now we have on the standby database server in the /u02/oradata/prac directory the Cold Backup, the Standby Control File and the initstdb.ora file.

Step 6 Setup the environment to support the Standby database

Create a password file for the “prac” standby database on the standby server:

orapwd file=orapwprac.ora entries=10

Configure the Oracle Net Services names on the primary and standby server:

Add the following two tnsnames.ora entries on the primary server and standby server.

 prac =

  (DESCRIPTION =

    (ADDRESS = (PROTOCOL = TCP)(HOST = linux1_vip)(PORT = 1522))

    (ADDRESS = (PROTOCOL = TCP)(HOST = linux2_vip)(PORT = 1522))

    (ADDRESS = (PROTOCOL = TCP)(HOST = linux3_vip)(PORT = 1522))

    (ADDRESS = (PROTOCOL = TCP)(HOST = linux4_vip)(PORT = 1522))

    (CONNECT_DATA =

      (SERVER = DEDICATED)

      (SERVICE_NAME = prac)     )   )

 stdb =

  (DESCRIPTION =

    (ADDRESS_LIST =

      (ADDRESS = (PROTOCOL = TCP)(HOST = linux5_vip)(PORT = 1521))

    )

    (CONNECT_DATA =

      (SERVER = DEDICATED)

      (SERVICE_NAME = prac)     )   )

 Check if the listeners are configured on both the primary and standby database servers.

Restart the listeners on the primary servers with:

[oracle@linux1 prac]$ . oraenv

ORACLE_SID = [prac1] ? crs

[oracle@linux1 prac]$ srvctl stop listener -n linux1

[oracle@linux1 prac]$ srvctl start listener -n linux1

[oracle@linux1 prac]$ srvctl stop listener -n linux2

[oracle@linux1 prac]$ srvctl stop listener -n linux3

[oracle@linux1 prac]$ srvctl stop listener -n linux4

[oracle@linux1 prac]$ srvctl start listener -n linux2

[oracle@linux1 prac]$ srvctl start listener -n linux3

[oracle@linux1 prac]$ srvctl start listener -n linux4

 For a non RAC database use the familiar listener commands:

lsnrctl stop

lsnrctl start

Specify the correct listener name if needed, check the listener.ora file for the name.

Step 7 Start the Physical Standby Database

Copy the standby control file to the controlfile names mentioned in the initstdb.ora file:

cp stdb.ctl /u02/oradata/prac/control01.ctl

cp stdb.ctl /u02/oradata/prac/control02.ctl

cp stdb.ctl /u02/oradata/prac/control03.ctl

 Set the correct environment variables and do the following:

 [oracle@linux5 prac]$ . oraenv

ORACLE_SID = [prac] ? prac

[oracle@linux5 prac]$ export ORACLE_SID=prac1

[oracle@linux5 prac]$ sqlplus /nolog

 SQL*Plus: Release 10.2.0.1.0 - Production on Tue Feb 28 11:19:33 2006

 Copyright (c) 1982, 2005, Oracle.  All rights reserved.

 SQL> conn / as sysdba

Connected to an idle instance.

SQL> create spfile from pfile='/u02/oradata/prac/initstdb.ora';

SQL> startup mount;

ORACLE instance started.

 Total System Global Area  524288000 bytes

Fixed Size                  1995688 bytes

Variable Size             213912664 bytes

Database Buffers          306184192 bytes

Redo Buffers                2195456 bytes

Database mounted.

 SQL> alter database recover managed standby database disconnect from session;

 Database altered.

 YES!!!

Step 8 Test the Physical Standby Database is performing properly

Force several log switches on the primary database w
ith:

alter system switch logfile;

 On the standby database connect as sysdba and check with;

select sequence#, applied, first_time, next_time, archived, status

from v$archived_log order by sequence#;

 SEQUENCE# APPLIED   FIRST_TIME      NEXT_TIME       ARCHIVED  STA

———- ——— ————— ————— ——— —

       426 YES       28-FEB-06       28-FEB-06       YES       A

       427 YES       28-FEB-06       28-FEB-06       YES       A

       428 YES       28-FEB-06       28-FEB-06       YES       A

       429 YES       28-FEB-06       28-FEB-06       YES       A

       430 YES       28-FEB-06       28-FEB-06       YES       A

       431 YES       28-FEB-06       28-FEB-06       YES       A

       432 NO        28-FEB-06       28-FEB-06       YES       A

       433 NO        28-FEB-06       28-FEB-06       YES       A

In the applied column you can see up to which logfile the standby database has catched up.

To see the role of the standby database do:

select database_role, db_unique_name, name from v$database;

 DATABASE_ROLE        DB_UNIQUE_NAME       NAME

——————– ——————– —————————

PHYSICAL STANDBY     stdb                 PRAC

  You can also check the file system to see that the archived log files really arrive on the standby destination site.

Troubleshoot tips

Always check the alert file on both the primary and the standby database site.

On Unix systems use tail –f alertprac.log to see the new messages appear at the end of the file.

 You might get errors like these:

Error 12514 received logging on to the standby

Tue Feb 28 13:07:22 2006

Errors in file /u01/sw1/app/oracle/admin/prac/bdump/prac1_arc5_11753.trc:

ORA-12514: TNS:listener does not currently know of service requested in connect descriptor

PING[ARC5]: Heartbeat failed to connect to standby 'stdb'. Error is 12514.

 OR

 Errors in file /u01/sw1/app/oracle/admin/prac/bdump/prac3_arcj_29779.trc:

ORA-12545: Connect failed because target host or object does not exist

PING[ARCj]: Heartbeat failed to connect to standby 'stdb'. Error is 12545.

 ==> These are Oracle Net Configuration errors, fix in listener.ora, tnsnames.ora or sqlnet.ora.

 Test the connectivity from the primary database to the standby with:

 [oracle@linux1 admin]$ sqlplus system/arnhem@stdb

 SQL*Plus: Release 10.2.0.1.0 - Production on Tue Feb 28 13:23:01 2006

 Copyright (c) 1982, 2005, Oracle.  All rights reserved.

 ERROR:

ORA-01033: ORACLE initialization or shutdown in progress

 Enter user-name:

THIS IS THE CORRECT MESSAGE YOU SHOULD GET SINCE THE STANDBY DATABASE IS IN MOUNT MODE!

or You might get the following error:

 Tue Feb 28 12:15:21 2006

Errors in file /u01/sw1/app/oracle/admin/prac/bdump/prac1_mrp0_31715.trc:

ORA-00313: open failed for members of log group 8 of thread 4

ORA-00312: online log 8 thread 4: '/u02/oradata/pracredo08.log'

ORA-27037: unable to obtain file status

Linux-ia64 Error: 2: No such file or directory

Additional information: 3

Clearing online redo logfile 8 /u02/oradata/pracredo08.log

Clearing online log 8 of thread 4 sequence number 334

Tue Feb 28 12:15:21 2006

&n
bsp;Thus:  /u02/oradata/pracredo08.log   There is a missing “/” between prac and redo08.log?

 Check the parameter log_file_name_convert: /u02/oradata/prac/, /u02/oradata/prac

AHA a Typo change it with:

alter system set log_file_name_convert = '/u02/oradata/prac/', '/u02/oradata/prac/' scope=spfile;

Note that you have to recycle the instance because the scope is spfile only!

So parameter settings are KEY!

Performance and Scalability Test
To test the performance and scalability of our 4 node RAC cluster we setup a test script with a trick we learned from Steve Adams. Use a common lock on a table, to kick-off all the sessions you want to have run concurrently on a system at once! We tested for 25, 50, 100, 200 and 400 concurrent users all firing the emp_dml.sql code at the same time.

The used performance scripts:

scott_acor.sql — Re-Creates the scott.emp table with 25 rows in it.

DROP TABLE EMP PURGE;

CREATE TABLE EMP
(EMPNO NUMBER(10) CONSTRAINT PK_EMP PRIMARY KEY,
ENAME VARCHAR2(10),
JOB VARCHAR2(9),
MGR NUMBER(4),
HIREDATE DATE,
SAL NUMBER(7,2),
COMM NUMBER(7,2),
DEPTNO NUMBER(2) CONSTRAINT FK_DEPTNO REFERENCES DEPT);

INSERT INTO EMP VALUES
(1,’SMITH’,'CLERK’,7902,to_date(’17-12-1980′,’dd-mm-yyyy’),800,NULL,20);
INSERT INTO EMP VALUES
(2,’ALLEN’,'SALESMAN’,7698,to_date(’20-2-1981′,’dd-mm-yyyy’),1600,300,30);
INSERT INTO EMP VALUES
(3,’WARD’,'SALESMAN’,7698,to_date(’22-2-1981′,’dd-mm-yyyy’),1250,500,30);
INSERT INTO EMP VALUES
(4,’JONES’,'MANAGER’,7839,to_date(’2-4-1981′,’dd-mm-yyyy’),2975,NULL,20);
INSERT INTO EMP VALUES
(5,’MARTIN’,'SALESMAN’,7698,to_date(’28-9-1981′,’dd-mm-yyyy’),1250,1400,30);
INSERT INTO EMP VALUES
(6,’BLAKE’,'MANAGER’,7839,to_date(’1-5-1981′,’dd-mm-yyyy’),2850,NULL,30);
INSERT INTO EMP VALUES
(7,’CLARK’,'MANAGER’,7839,to_date(’9-6-1981′,’dd-mm-yyyy’),2450,NULL,10);
INSERT INTO EMP VALUES
(8,’SCOTT’,'ANALYST’,7566,to_date(’13-JUL-1987′)-85,3000,NULL,20);
INSERT INTO EMP VALUES
(9,’KING’,'PRESIDENT’,NULL,to_date(’17-11-1981′,’dd-mm-yyyy’),5000,NULL,10);
INSERT INTO EMP VALUES
(10,’TURNER’,'SALESMAN’,7698,to_date(’8-9-1981′,’dd-mm-yyyy’),1500,0,30);
INSERT INTO EMP VALUES
(11,’ADAMS’,'CLERK’,7788,to_date(’13-JUL-1987′)-51,1100,NULL,20);
INSERT INTO EMP VALUES
(12,’JAMES’,'CLERK’,7698,to_date(’3-12-1981′,’dd-mm-yyyy’),950,NULL,30);
INSERT INTO EMP VALUES
(13,’FORD’,'ANALYST’,7566,to_date(’3-12-1981′,’dd-mm-yyyy’),3000,NULL,20);
INSERT INTO EMP VALUES
(14,’MILLER’,'CLERK’,7782,to_date(’23-1-1982′,’dd-mm-yyyy’),1300,NULL,10);
INSERT INTO EMP VALUES
(15,’MARTIN’,'SALESMAN’,7698,to_date(’28-9-1991′,’dd-mm-yyyy’),1250,1400,30);
INSERT INTO EMP VALUES
(16,’BLAKE’,'MANAGER’,7839,to_date(’1-5-1991′,’dd-mm-yyyy’),2850,NULL,30);
INSERT INTO EMP VALUES
(17,’CLARK’,'MANAGER’,7839,to_date(’9-6-1991′,’dd-mm-yyyy’),2450,NULL,10);
INSERT INTO EMP VALUES
(18,’SCOTT’,'ANALYST’,7566,to_date(’13-JUL-1997′)-85,3000,NULL,20);
INSERT INTO EMP VALUES
(19,’ALLEN’,'SALESMAN’,7698,to_date(’20-2-1991′,’dd-mm-yyyy’),1600,300,30);
INSERT INTO EMP VALUES
(20,’TURNER’,'SALESMAN’,7698,to_date(’8-9-1991′,’dd-mm-yyyy’),1500,0,30);
INSERT INTO EMP VALUES
(21,’ADAMS’,'CLERK’,7788,to_date(’13-JUL-1997′)-51,1100,NULL,20);
INSERT INTO EMP VALUES
(22,’JAMES’,'CLERK’,7698,to_date(’3-12-1991′,’dd-mm-yyyy’),950,NULL,30);
INSERT INTO EMP VALUES
(23,’FORD’,'ANALYST’,7566,to_date(’3-12-1991′,’dd-mm-yyyy’),3000,NULL,20);
INSERT INTO EMP VALUES
(24,’MILLER’,'CLERK’,7782,to_date(’23-1-1992′,’dd-mm-yyyy’),1300,NULL,10);
INSERT INTO EMP VALUES
(25,’SMITH’,'CLERK’,7902,to_date(’17-12-1990′,’dd-mm-yyyy’),800,NULL,20);

lock_emp.sql — Locks the common table.

prompt Performance Lock Script
prompt
prompt Starts with: delete from scott.emp;
delete from scott.emp;
Prompt
prompt Now start the other SQL*PLus sessions
prompt
prompt Ready for take off?
prompt A rollback statement will be issued and the locks on the rows are freed
pause
rollback;
col syscol format a20 heading “Start Time of Test”
select to_char(sysdate, ‘YYYYMMDD HH24MISS’) syscol from dual;

emp_dml.sql — Fires a heavy select statement and does an insert of 5000 rows in the emp table. Copy the emp table to the emp_org table once (With 25 rows) for the tests!

set lin 300 verify off trimspool on feedback off
col spoolfile noprint new_value _spoolfile
select ‘log/’||INSTANCE_NAME||’_'||to_char(25+1 + &1.*5000)||’_tot_’||to_char(25+5000 + &1.*5000)||’.txt’ spoolfile from sys.v_$instance;
spool &_spoolfile
update emp
set sal = sal + 1
where job = ‘PRESIDENT’;
rollback;
set timing on
select count(*) COUNT_1A from emp_org a, emp_org b, emp_org c, emp_org d, emp_org e
;
begin
for n in 1 .. 5000 loop
INSERT INTO EMP VALUES (n+25 + &1.*5000,’SMITH’,'CLERK’,7902, trunc(sysdate -36500),800,NULL,20);
update emp set EMPNO = n+25 + &1.*5000 where EMPNO = n+25 + &1.*5000;
end loop;
end;
/
commit;
spool off
exit;

The kick_off_.sh shell scripts looked like:

#!/bin/bash
export ORACLE_HOME=/u00/sw1/app/oracle/product/102
PATH=$ORACLE_HOME/bin:.

exec sqlplus -s scott/tiger@acor @emp_dml 1 &
exec sqlplus -s scott/tiger@acor @emp_dml 2 &
exec sqlplus -s scott/tiger@acor @emp_dml 3 &
exec sqlplus -s scott/tiger@acor @emp_dml 4 &
exec sqlplus -s scott/tiger@acor @emp_dml 5 &
exec sqlplus -s scott/tiger@acor @emp_dml 6 &
## etc. etc. etc. up to the number of concurrent sessions you wanted to test.

Test results:

The elapsed time is mentioned in seconds.

We got the best scalability on the 2 node RAC database which was running on the 4 CPU systems (linux1 and linux2). Our test on the 4 node RAC database with respectively 4,4,2 and 1 CPU’s showd that for 400 concurrent users, the single and dual CPU systems were causing the slowdown. The load balancer gives these system more processes because default the single and dual CPU systems have less Unix processes running on the server. Between a 4 CPU and single CPU system you can have easily a difference of 50 Unix processes.

We did not test the performance by tweaking the load balancer.

We can however conclude that the 2 Node RAC cluster with 4 CPU’s gives a very scalable solution. It can handle twice as many transactions as a single node 4 CPU system during heavy (400 concurrent users) load.

Best practice: To prevent load balancing mismatches it is the easiest to ask for servers with the same specification. Because of the fact that the interconnect traffic is CPU intensive you should not use single CPU servers in a RAC cluster.

If you are using RHEL or SUSE Linux, then you must install the cvuqdisk package. Without cvuqdisk the Cluster Verify Utility is unable to discover shared disks, and you receive the error message “Package cvuqdisk not installed�?. Install this rpm from the Oracle Clusterware Installation Disk.

Go to the clusterware/rpm directory and install it with:

rpm -iv cvuqdisk-1.0.1-1.rpm

Unpack the Oracle 10gR2 Clusterware software with:
cpio -idcmv < 10201_clusterware_linuxitanium.cpio

From the clusterware software run the Cluster Verification Utility (runcluvfy.sh) which is in the cluvfy directory.

We do three pre installation checks with the CVU:
- Performing post-checks for hardware and operating system setup
- Verifying system requirement
- Performing pre-checks for cluster services setup

The commands we used are:

./runcluvfy.sh stage -post hwos -n linux1,linux2,linux3,linux4 -verbose

./runcluvfy.sh comp sys -n linux1,linux2,linux3,linux4 -p crs -osdba dba -orainv oinstall -verbose

./runcluvfy.sh stage -pre crsinst -n linux1,linux2,linux3,linux4 -verbose

When you passed all checks its finally time to start an Oracle Universal Installer (OUI).

Go to the clusterware directory as oracle and start the OUI with ./runInstaller.
Specify the oraInventory, you can only use one for the cluster!! And click Next.
Specify correct Path. Click Next, (Here you can add Product Languages if you want!!).

When you've done you're pre-checks and have run all runcluvy.sh scripts successful you should have all checks OK. (If not look at the error message, fix it, click on retry and hope for success!)

Accept the Cluster name (Default crs). We changed the default private node name linux-priv to you're /etc/hosts interconnect private IP address name. Adjusted the Virtual host name (From linux3-vip to linux3_vip) according to /etc/hosts as well and went on.

We repeated the steps for al our four nodes.

Then we adjusted the public (10.17.246.0) and private (16.0.0.0 Non Routable!) subnets.

At the specify OCR Location screen we chose Normal Redundancy (This means having two OCR's).
At the specify Voting Disk Location screen we chose Normal Redundancy (This means having three Voting Disks).

Finally the Summary Screen Appeared, and it was time to do the Install.

At the end of the install you have to run orainstRoot.sh on all four nodes.
!!!DO NOT TRY TO RUN THE SCRIPTS SIMULTANEOUSLY!!!
This could corrupt the OCR.

After this you have to run the root.sh script on all nodes this formats the OCR and starts the CSS Deamons:

[root@linux3 crs]# ./root.sh
WARNING: directory '/u01/sw1/app/oracle/product/10.2.0' is not owned by root
WARNING: directory '/u01/sw1/app/oracle/product' is not owned by root
WARNING: directory '/u01/sw1/app/oracle' is not owned by root
Checking to see if Oracle CRS stack is already configured

Setting the permissions on OCR backup directory
Setting up NS directories
Oracle Cluster Registry configuration upgraded successfully
WARNING: directory '/u01/sw1/app/oracle/product/10.2.0' is not owned by root
WARNING: directory '/u01/sw1/app/oracle/product' is not owned by root
WARNING: directory '/u01/sw1/app/oracle' is not owned by root
Successfully accumulated necessary OCR keys.
Using ports: CSS=49895 CRS=49896 EVMC=49898 and EVMR=49897.
node :
node 1: linux3 linux3_rac linux3
node 2: linux1 linux1_rac linux1
node 3: linux2 linux2_rac linux2
node 4: linux4 linux4_rac linux4
Creating OCR keys for user ‘root’, privgrp ‘root’..
Operation successful.
Now formatting voting device: /u01/cd1/css/CSSFile
Now formatting voting device: /u02/cd2/css/CSSFile
Now formatting voting device: /u03/cd3/css/CSSFile
Format of 3 voting devices complete.
Startup will be queued to init within 90 seconds.
Adding daemons to inittab
Expecting the CRS daemons to be up within 600 seconds.
CSS is active on these nodes.
linux3
CSS is inactive on these nodes.
linux1
linux2
linux4
Local node checking complete.

Run root.sh on remaining nodes to start CRS daemons on the other nodes.

After this go back to the OUI screen and continue the post-installation steps.
The OUI starts the VIPCA. Fill in the network configuration details and continue.

Finally we got CRS on the 4 node cluster up and running.

In our next blog we show how we created a 4 node RAC database on our cluster.

Download the appropriate RPM’s. For the kernel module, download the one that matches the
distribution, platform, kernel version and the kernel flavor (smp, hugemem, psmp, etc). For tools, simply match the platform and distribution. The tools RPM’s are split into two. ocfs2tools includes the command line tools whereas ocfs2console includes the GUI front end for the tools. One does not have to install the console, though it is recommended for ease of use. Install the RPM’s using the rpm install or rpm upgrade command.

We downloaded the most recent OCFS2 rpm’s for Itanium (ia64).
OCFS2 Kernel Module: ocfs2-2.6.9-22.0.1.EL-1.0.9-1.ia64.rpm
OCFS2 Console: ocfs2console-1.0.3-1.ia64.rpm
OCFS2 Tools: ocfs2-tools-1.0.3-1.ia64.rpm

Because we installed ocfs2-2.6.9-22.0.1.EL we needed to patch RHEL4 U2 to
kernel-2.6.9-22.0.1.EL.ia64.rpm to make it work.

Installation of the RHEL4 U2 patch was as easy as doing:
rpm -ivh kernel-2.6.9-22.0.1.EL.ia64.rpm

Install OCFS2 on Linux1 to Linux5 as root with:

rpm -Uvh ocfs2-2.6.9-22.0.1.EL-1.0.9-1.ia64.rpm \
ocfs2console-1.0.3-1.ia64.rpm \
ocfs2-tools-1.0.3-1.ia64.rpm

Preparing… ########################################### [100%]
1:ocfs2-tools ########################################### [ 33%]
2:ocfs2-2.6.9-22.0.1.EL ########################################### [ 67%]
3:ocfs2console ########################################### [100%]

Check existence of ocfs2 on a node with:

which ocfs2console The output will be: /usr/sbin/ocfs2console

To configure OCFS2 start the ocfs2console. Since this is a GUI you should have a window that can run the xclock. We simply start vncserver and connect from our laptop with VNC Viewer.

Starting the vncserver is done by typing: vncserver.
To stop a running vncserver do: vncserver -kill linux1:1
Always test you’re DISPLAY output with: xclock.

Now start the ofcs2console GUI tool as root with: ocfs2console &
Select [Cluster] –> [Configure Nodes]. This starts the OCFS Cluster Stack, and brings up the “Node Configuration�? dialog. Click the [Add] button, This brings up the “Add Node�? dialog.
Enter the node name and IP address node for node. Click on Apply and Click on [Cluster] Propagate Cluster Configuration, you see the next screen, then do close and exit.

Now you have gotten a /etc/ocfs2/cluster.conf file. This file should be the same on all cluster nodes. Verify if the /etc/ocfs2/cluster.conf file is propagated to all the cluster nodes.

Content of our cluster.conf file:

[root@linux1 ocfs2]# more cluster.conf
node:
ip_port = 7777
ip_address = 10.17.246.1
number = 0
name = linux1
cluster = ocfs2

node:
ip_port = 7777
ip_address = 10.17.246.2
number = 1
name = linux2
cluster = ocfs2

node:
ip_port = 7777
ip_address = 10.17.246.3
number = 2
name = linux3
cluster = ocfs2

node:
ip_port = 7777
ip_address = 10.17.246.4
number = 3
name = linux4
cluster = ocfs2

cluster:
node_count = 4
name = ocfs2

Before we can do anything with OCFS2 like formatting or mounting the file system, we need to first have OCFS2’s cluster stack, O2CB, running (It should run after you created the cluster.conf file). The stack includes the following services:
- NM: Node Manager that keep track of all the nodes in the cluster.conf
- HB: Heart beat service that issues up/down notifications when nodes join or leave the cluster
- TCP: Handles communication between the nodes
- DLM: Distributed lock manager that keeps track of all locks, its owners and status
- CONFIGFS: User space driven configuration file system mounted at /config
- DLMFS: User space interface to the kernel space DLM
All of the above cluster services have been packaged in the O2CB system service (/etc/init.d/o2cb).

To check the status do:
/etc/init.d/o2cb status
Module “configfs”: Loaded
Filesystem “configfs”: Mounted
Module “ocfs2_nodemanager”: Loaded
Module “ocfs2_dlm”: Loaded
Module “ocfs2_dlmfs”: Loaded
Filesystem “ocfs2_dlmfs”: Mounted
Checking cluster ocfs2: Online
Checking heartbeat: Active
If the modules are not loaded, load them with:
/etc/init.d/o2cb load
To start the created OCFS2 cluster do:
/etc/init.d/o2cb online ocfs2
You can also offline the cluster and unload the modules but we don’t need that now.
You need to configure the on-boot properties of the O2BC driver, so that the cluster stack will start on each boot of the server. Do this with:
/etc/init.d/o2cb configure Answer yes, and give your clustername: ocfs2 (is the default).

Now format the pre-created devices, from 1 node (linux1 do):
mkfs.ocfs2 -b 4K -C 32K -N 4 -L oracrs1 /dev/sdc
mkfs.ocfs2 -b 4K -C 32K -N 4 -L oracrs2 /dev/sdd
mkfs.ocfs2 -b 4K -C 32K -N 4 -L oracrs3 /dev/sde
mkfs.ocfs2 -b 4K -C 32K -N 4 -L oraclehome1 /dev/sdf
mkfs.ocfs2 -b 4K -C 32K -N 4 -L oradlehome2 /dev/sdg
mkfs.ocfs2 -b 4K -C 128K -N 4 -L oradata /dev/sdh
mkfs.ocfs2 -b 4K -C 128K -N 4 -L oraflra /dev/sdi

We used 4K block sizes and 32k or 128K cluster sizes. From the Frequently Asked Questions we got the tip to use 4k block sizes (The block size can be smaller 2k, 1k or 512bytes), for datafiles it was advised to use a bigger cluster size.

When the devices are created we can mount them on all cluster nodes. Be aware of the different device names on linux4 node!

mount :t ocfs2 datavolume,nointr /dev/sdc /u01/cd1
mount :t ocfs2 datavolume,nointr /dev/sdd /u02/cd2
mount :t ocfs2 datavolume,nointr /dev/sde /u03/cd3
mount :t ocfs2 /dev/sdf /u01/sw1
mount :t ocfs2 /dev/sdg /u09/sw2
mount :t ocfs2 datavolume,nointr /dev/sdh /u02/oradata
mount :t ocfs2 datavolume,nointr /dev/sdi /u03/oraflra

Oracle database users must mount any volume that will contain the Voting Disk file, Cluster Registry (OCR), Data files, Redo logs, Archive logs, and Control files with the datavolume mount option so as to ensure that the Oracle processes open the files with the o_direct flag.

From the OCFS2 Frequently Asked Questions (FAQ) we read that option “nointr�? is needed for datavolume mountpoints as well. The “nointr�? option ensures that IO’s are not interrupted by signals.
Any other type of volume, including an Oracle home, should not be mounted with this mount option.
ALL mountpoints need to have the _netdev option as well. This is a MUST for OCFS2 volumes. The “_netdev�? option indicates that the volume is to be mounted AFTER the network is started and dismounted BEFORE the network is shutdown.

If all mountpoints are mounted OK, you can edit the fstab to make the filesystems available after a reboot.

Here is a list of the /etc/fstab file from the linux1 server.

[root@linux1 bin]# cat /etc/fstab
# This file is edited by fstab-sync - see ‘man fstab-sync’ for details
/dev/VolGroup00/LogVol00 / ext3 defaults 1 1
/dev/cciss/c0d0p1 /boot/efi vfat defaults 0 0
none /dev/pts devpts gid=5,mode=620 0 0
none /dev/shm tmpfs defaults 0 0
none /proc proc defaults 0 0
none /sys sysfs defaults 0 0
/dev/VolGroup00/LogVol01 swap swap defaults 0 0
/dev/sdc /u01/cd1 ocfs2 _netdev,datavolume,nointr 0 0
/dev/sdd /u02/cd2 ocfs2 _netdev,datavolume,nointr 0 0
/dev/sde /u03/cd3 ocfs2 _netdev,datavolume,nointr 0 0
/dev/sdf /u01/sw1 ocfs2 _netdev 0 0
/dev/sdg /u09/sw2 ocfs2 _netdev 0 0
/dev/sdh /u02/oradata ocfs2 _netdev,datavolume,nointr 0 0
/dev/sdi /u03/oraflra ocfs2 _netdev,datavolume,nointr 0 0
/dev/hda /media/cdrom auto pamconsole,exec,noauto,managed 0 0

Create a test file on the shared storage and verify if every cluster node can see the new file.

Test if the mountpoints come up correctly after a reboot of all the cluster nodes.

Pre-installation Steps

Check the OS
On the HP Demo Servers RHEL3 or Windows 2003 was installed. (HP Integrity Servers can run several different Operating systems, with VM technology even at the same time!).
All servers need to have the same OS. To work with OCFS2 we have to have RHEL4. HP used the RHEL4 Update 2 install CD’s and completely Re-Installed the OS on all 6 servers. For linux6 the AMD RHEL4 Update 2 distribution was used. (Check you’re OS version with “uname :a).

Common RHEL4 installer prompts
- Remove all partitions (Yes)
- Fill in hostname + IP address
- Fill in IP address for eth0 and eth1 + Check the box automatic start on reboot.
- Fill in the gateway (we used 10.17.246.254).
- Use no firewall
- Language English
- Disable SELinux
- Set the root password
- Customize software packages (Remove GNOME, Add KDE, Remove Windows File Server, Add Legacy software, Add Development tools).
Post Install steps:
- Fill in license info later
- Set the time zone (Amsterdam) and time.

Look at available system tools with “system-config {+ Hit the TAB Key, twice}.
For instance to start the network GUI on RHEL4 to change eth0 and eth1 settings do:
system-config-network

Other RHEL4 info can be found in the Red Hat Manuals: http://www.redhat.com/docs/manuals/

Perform Network Configuration on all Nodes
Setup /etc/hosts on all nodes. Test the connectivity with “ping hostname�?. To be able to connect to all the HP servers from within the LogicaCMG network we had to ask for an IP range via a change at our Central IT organization. We are allowed to use the 10.17.246.x range during our PoC.

For the RAC Interconnect we decided to use a non-routable IP range 16.0.0.x, this was necessary to make sure the eth0 and eth1 network cards were addressed correctly.
The public and private IP addresses must be in different subnets. E.g. 10.17.246.x and 10.17.247.x would have been OK as well.

Create oracle user and directories
On Linux you first have to create the groups and then you create the user.
It is essential that on ALL cluster nodes the group ID’s are identical.

Create the groups as root on all servers with:
groupadd -g 200 dba
groupadd -g 201 oinstall

Create the oracle user as root on all servers with (and watch out for Tabs in scripts):
useradd -u 202 -g 201 -G 200 \
-d /home/oracle \
-s /bin/bash \
-c “Oracle Software Owner” \
-p oracle oracle

Change the password of the oracle user as root on all servers with:
passwd oracle

Create the CORDS / OFA directories:
mkdir -p /u01/cd1
mkdir -p /u02/cd2
mkdir -p /u03/cd3

mkdir -p /u01/sw1
mkdir -p /u09/sw2

mkdir -p /u02/oradata
mkdir -p /u03/oraflra

chown -R oracle:dba /u02/oradata
chown -R oracle:dba /u03/oraflra

Create partitions on shared storage
For Itanium servers it is necessary to have a management console to configure the MSA1500 Storage Area Network. We ran the HP Storage Array Control Utility (ACU 7.x) on the AMD server which could act as a management console during the PoC.

We asked the HP Engineer to carve the disks according to our prepared standards.
(See our previous PoC Blog : Preparation Steps).

It is essential to know and administer, how every node sees the carved devices.
Mounting different devices to different mountpoints could corrupt you’re data. Management of the /etc/fstab file is critical, DO NOT “scp ” the fstab file to all the other nodes, when you have different device names on different nodes like we had.

During OCFS2 mount you have to make sure every node mounts the same device to the same mountpoint. Different device names are for instance due to different number of local disks or different number of array controllers in local servers.

Configure the Linux Servers for oracle
Adjust Linux Kernel settings to run Oracle database SGA properly.

We added the following kernel settings in /etc/sysctl.conf on all 6 server nodes.

kernel.shmall = 2097152
kernel.shmmax = 2147483648
kernel.shmmni = 4096
kernel.sem = 250 32000 100 128
fs.file-max = 65536
net.ipv4.ip_local_port_range = 1024 65000
net.core.rmem_default = 262144
net.core.rmem_max = 262144
net.core.wmem_default = 262144
net.core.wmem_max = 262144

To make the changes active you have to reboot, this can easily be done as root with:
shutdown :r now

Configure the hangcheck-timer Kernel Module
The hangcheck-timer.ko file is missing on the Itanium servers. Metalink shows that it should be part of the RHEL4 distribution. After some further searches, this turns out the be a known issue, documented in oracle bug 4714727. There are re-kitted kernels available to fix this issue, either from Oracle or from Redhat. In case you contact Oracle, log a service request to get a re-kitted kernel as described in the bug 4714727. In case you contact Redhat, the Red Hat issue 79368 should be referenced, this is the corresponding issue for the hangcheck-timer problem.

Because it took very long to find this issue we had no time to test the installation of the hangcheck-timer. We urge customers to use a hangcheck-timer in their production environments.

Configure RAC Nodes for Remote Access
It is advised to setup secure shell (ssh) for the oracle user on all cluster nodes.

Here are our steps. Logon as the oracle user on all 4 nodes and do:

cd ~
pwd
touch ~/.ssh/authorized_keys
cd ~/.ssh

# Generate the RSA and DSA public and private keys with:
/usr/bin/ssh-keygen -t rsa (Leave the pass phrase empty by pressing the ENTER key)
/usr/bin/ssh-keygen -t dsa (Leave the pass phrase empty by pressing the ENTER key)

# from the .ssh dir, do a cat of the local and remote keys..
ssh linux1 cat /home/oracle/.ssh/id_rsa.pub >> authorized_keys
ssh linux1 cat /home/oracle/.ssh/id_dsa.pub >> authorized_keys
ssh linux2 cat /home/oracle/.ssh/id_rsa.pub >> authorized_keys
ssh linux2 cat /home/oracle/.ssh/id_dsa.pub >> authorized_keys
ssh linux3 cat /home/oracle/.ssh/id_rsa.pub >> authorized_keys
ssh linux3 cat /home/oracle/.ssh/id_dsa.pub >> authorized_keys
ssh linux4 cat /home/oracle/.ssh/id_rsa.pub >> authorized_keys
ssh linux4 cat /home/oracle/.ssh/id_dsa.pub >> authorized_keys

# verify: 4 nodes times 2 lines per node gives 8 lines with “wc :l ” (word count lines)
wc :l authorized_keys

# distribute the authorized_keys file to the other cluster nodes with scp (secure copy).
scp authorized_keys linux2:/home/oracle/.ssh/
scp authorized_keys linux3:/home/oracle/.ssh/
scp authorized_keys linux4:/home/oracle/.ssh/

Test oracle user equivalence by issuing the following 4 commands on all 4 nodes:
ssh linux1 date
ssh linux2 date
ssh linux3 date
ssh linux4 date
All commands should give the date without prompting for a password!
If you did not check it correctly, the Cluster Verify Utility (runcluvfy.sh) will warn you.

For ease of maintenance it is very nice to setup user equivalence for the root user as well.
This is however not an installation requirement.

Due to the fact that the OUI uses rcp (remote copy) and rsh (remote shell) it is also necessary to setup host equivalence, at least during the usage of the OUI. After the installation you can disable it, but you should not forget to set it back again when you are going to apply a patch with the OUI.

For rsh to work two rsh RPM’s should be installed on each node:
rpm :q rsh rsh-server
rsh-0.17-25.3
rsh-server-0.17-25.3

If they are not on the server install them from the installation media with:
su :
rpm :ivh rsh-0.17-25.3.ia64.rpm rsh-server-0.17-25.3.ia64.rpm

To enable the “rsh�? and “rlogin�? services, the “disable�? attribute in the /etc/xinetd.d/rsh file must be set to “no�? and xinitd must be reloaded on all nodes of the cluster with:

su :
chkconfig rsh on
chkconfig rlogin on
service xinted reload

To allow the “oracle�? Unix user account to be trusted among RAC nodes, create the /etc/hosts.equiv file on all cluster nodes with:

su :
touch /etc/hosts.equiv
chmod 600 /etc/hosts.equiv
chown root:root /etc/hosts.equiv

Edit the file like this:
[root@linux1 ~]# cat /etc/hosts.equiv
linux1 root
linux2 root
linux3 root
linux4 root
linux1 oracle
linux2 oracle
linux3 oracle
linux4 oracle
linux1_rac oracle
linux2_rac oracle
linux3_rac oracle
linux4_rac oracle

Before attempting to test rsh, ensure that you are using the correct version of rsh. By default, RHEL4 puts /usr/Kerberos/sbin at the head of the $PATH variable. Rename the Kerberos scripts to force the system to use the /usr/sbin/rsh command.

mv /usr/kerberos/bin/rsh /usr/kerberos/bin/rsh.original
mv /usr/kerberos/bin/rcp /usr/kerberos/bin/rcp.original
mv /usr/kerberos/bin/rlogin /usr/kerberos/bin/rlogin.original

Test the host equivalence with the following commands on all 4 cluster nodes:

rsh linux1 ls :l /etc/hosts.equiv
rsh linux2 ls :l /etc/hosts.equiv
rsh linux3 ls :l /etc/hosts.equiv
rsh linux4 ls :l /etc/hosts.equiv
All commands should give the listing without prompting for a password!

Check required RPM Packages for Oracle 10gR2
Required RPM packages for RHEL4 for correct working of 10gR2
(See Page 2-28 CRS Install Guide Part: B14203-05).

binutils-2.15.92.0.2-10.EL4
compat-db-4.1.25-9
control-center-2.8.0-12
gcc-3.4.3-9.EL4
gcc-c++-3.4.3-9.EL4
glibc-2.3.4-2
glibc-common-2.3.4-2
gnome-libs-1.4.1.2.90-44.1
libstdc++-3.4.3-9.EL4
libstdc++-devel-3.4.3-9.EL4
make-3.80-5
pdksh-5.2.14-30
sysstat-5.0.5-1
xscreensaver-4.18-5.rhel4.2

Check these RPM’s with: rpm :q as follows:

rpm -q binutils
rpm -q compat-db
rpm -q control-center
rpm -q gcc
rpm -q gcc-c++
rpm -q glibc
rpm -q glibc-common
rpm -q gnome-libs
rpm -q libstdc++
rpm -q make-3.80-5
rpm -q pdksh
rpm -q sysstat
rpm -q xscreensaver

Final pre-checks
Make sure that the ForwardX11 attribute is set to “no�? for oracle

touch ~/.ssh/config

Edit the just created config file; fill in:

Host *
ForwardX11 no

Do this once then do on copy to all other nodes with:
scp /home/oracle/.ssh/config linux2:/home/oracle/.ssh/config
scp /home/oracle/.ssh/config linux3:/home/oracle/.ssh/config
scp /home/oracle/.ssh/config linux4:/home/oracle/.ssh/config
scp /home/oracle/.ssh/config linux5:/home/oracle/.ssh/config

Modify ~/.bash_profile (again on all nodes) add:
# prevent stty errors ora-installer
if [ -t 0 ]; then
stty intr ^C
fi

Do this once then do on all other nodes with:
scp .bash-profile linux2:/home/oracle/.bash_profile
scp .bash-profile linux3:/home/oracle/.bash_profile
scp .bash-profile linux4:/home/oracle/.bash_profile
scp .bash-profile linux5:/home/oracle/.bash_profile

Oracle user should have Umask=022 or 0022,
check with: umask

Disk space should be at least 8Gb for 1 CRS+ 1 ASM + 1RAC Home.
and /tmp should have enough (>> 400Mb) space:
check with df :k.

RAM should be at least 1Gb
check with: dmesg | grep “Memory”

Swap space should be Equal to size of RAM (for RAM >= 2Gb).
Check with top: or “grep SwapTotal /proc/meminfo�?

For this PoC we added 4, 8 or 16Gb of swap space with:
cd /tmp
dd if=/dev/zero of=tempswap bs=1k count=4000000
chmod 600 tempswap
mke2fs tempswap
mkswap tempswap
swapon tempswap

(for 8Gb we used count=8000000 and for 16Gb we used count=16000000).

Set the date correct on the cluster nodes:
Normally you should use a Network Time Protocol Server.

For the PoC we used: date 0126133006 {MMDDHHMIYY}, on all 4 cluster nodes,
this command is not allowed with ssh

Check the processor info
Check with: cat /proc/cpuinfo (Itanium is used)
Match this with the Oracle software we already prepared that.

Increase shell limits for oracle,
Do this by adding lines to /etc/security/limits.conf file
oracle soft nproc 2047
oracle hard nproc 16384
oracle soft nofile 1024
oracle hard nofile 65536

Edit the file /etc/pam.d/login :
Add following if it does not already exist.
session required pam_limits.so

Edit the /etc/profile (for BASH users)
if [ $USER = "oracle" ]; then
if [ $SHELL = "/bin/ksh" ]; then
ulimit -p 16384
ulimit -n 65536
else
ulimit -u 16384 -n 65536
fi
fi

Distribute the last two local (/etc files to all cluster nodes).

We finished the pre-installation Steps.

In the next blog we discuss how to intall and configure OCFS2.

Preparation Steps

To prepare ourselves we have read the paper of Jeffrey Hunter: Build Your Own RAC 10g Release 2 Cluster on Linux and Firewire http://www.oracle.com/technology/pub/articles/hunter_rac10gr2.html

This paper has been in the top 5 most read DBA Articles on OTN for the last 3 months. And it is one of the first who has worked with OCFS2. Since we want to setup a “Shared Everything�? environment on Linux, we want to use the Open Source Oracle Clustered File System software. Although this software is not certified yet with Oracle 10gR2, we think it can become a very important software component for RAC databases on Linux systems.

Other documents that are a Must Read are the Install and Deployment guides from Oracle:
Oracle Clusterware and Oracle Real Application Clusters Installation Guide 10g Release 2 (10.2) for Linux. Part number: B14203-05 http://otn.oracle.com

Oracle Clusterware and Oracle Real Application Clusters Administration and Deployment Guide 10g Release 2 (10.2). Part number: B14197-03 http://otn.oracle.com

We spend some time preparing ourselves on a white-board, and come up with an Instance/Server Matrix:

To be able to run a RAC database on 4 nodes all cluster nodes need two public and one private IP address. Because we try to implement MAA we setup Network for two RAC clusters.

Network Setup for cluster 1 named crs:

Network Setup for cluster 2 (Standby Cluster) named crs2:

The Load Generator node has two network cards that are configured as:

Note that the 16.0.0.x IP addresses are Non Routable. You need to use different IP subnets for you’re eth0 and eth1 IP ranges to make sure the correct network cards are used.

The easiest setup for the network cards is during the initial installation of RHEL4 U2. You can also specify that both network interfaces should be brought up after a reboot of the server.

Setup this configuration in you’re /etc/hosts file on each node. And do ping tests to test connectivity. Check the network card setup with “ifconfig :a�? on each node.

Fiber channel connections are made between Linux 1 to Linux 6 to the HP San Switch on the respective port numbers 1 to 6.

From the Linux 1 to Linux 6 LAN cables from (Port A) to port 1 to 6 on the Pro Curve switch.
And Port B on ports 11 to 16.

Storage Layout
All devices are setup on the MSA1500 with RAID 1+0. From the 8 * 73 Gb Bruto we can allocate Netto 4*73Gb = 292Gb. It is important to carve the disks into devices with correct sizes so that we can optimally use the available storage. As DBA’s we are responsible to give the correct carving instructions. We asked for the same sizes for both clusters.

It is important to notice that the names of the devices can be different on each node. Check the devices per node as root with: fdisk :l ; and fill in a table like we did.

Applied CORDS (LogicaCMG’s Remote DBA Standards) : OFA Rules:
Mountpoint’s first part /u0x (should be) a physical disk (or array of disks).
Second part is a functional specification.

cd = cluster data
sw = software
oradata = oracle database data
oraflra = oracle flash recovery area

Unused space left on MSA1500 is 28975 Mb Bruto.

Devices are also known as LUN’s.

ASM Disks do not need to be mounted, they are raw disks formatted and maintained by the +ASM1, +ASM2, +ASM3, and +ASM4 instances.

Disclaimer: In a production environment you should always have you’re standby database on a different SAN. For our PoC we had to do the tests with one SAN.

CORDS / OFA Setup
As best practice we use the CORDS / OFA installation guidelines to configure the cluster.

Starting with Oracle Database 10g Release 2 (10.2), Oracle Clusterware should be installed in a separate Oracle Clusterware home directory, because succeeding versions of Oracle Clusterware will overwrite the Oracle Clusterware installation in the same path. This is also advised for ASM. To install ASM in its own directory you have to run the database OUI twice. The first time you run the database OUI, install the ASM home. The second time you run the database OUI, install the database with RAC home. This allows you to patch the different homes (Clusterware/ASM/RAC) separately.

Starting with Oracle Database 10g Release 2 (10.2) with RAC, Oracle Clusterware provides for the creation of a mirrored Oracle Cluster Registry (OCR) file, enhancing cluster reliability. And you can configure the Cluster Synchronization Services (CSS) with multiple voting disks. In Oracle 10g Release 1 (10.1) you could only configure one voting disk. To obtain benefits of the multiple voting disks you must configure at least three voting disks on independent shared physical disks. It is possible (but not recommended by Oracle) to use RAW devices for these files.

Note that Oracle Clusterware is also available for use with single-instance databases and applications that you deploy on clusters. The API libraries required to use with single-instance databases are provided with the Oracle Client installation media.

We tried to specify different Oracle Inventories for CRS and RAC but the RAC installer does not recognize the cluster when the oraInventory points to an empty directory. You have to specify the CRS_HOME inventory to do a RAC installation.

Oracle recommends that you use Automatic Storage Management (ASM) for Database and Recovery files. LogicaCMG prefers the use of a clustered file system, for ease of maintenance.

Note if you create more then one database on the cluster, then they all share the same OCR.
Note that with Oracle Database 10g Release 2 (10.2), Cluster Ready Services, or CRS, is now called Oracle Clusterware.

On the clustered file system the following CORDS / OFA directory standards are used:

Note that if you’re Oracle home is not on a shared file system, then OUI propagates the software onto the local disks of the other nodes with “rcp’. The OUI also uses rcp for copying files to the local /etc directory. So you always have to setup you’re /etc/hosts.equiv files.

In the next Blog we are going to write about the pre-installation steps.

Hardware Inventory:
Hardware Inventory
5 HP-RX4620 Itanium Servers (2 with 4 CPUs, 1 with 2CPUs and 2 with 1 CPU).
The 4 CPU and 2 CPU servers have a processor speed of 1,6Ghz; the 1 CPU servers have processor speed of 1,5Ghz. Four servers have 2 * 73Gb local storage and one server has 2 * 36Gb local storage. The 4 CPU servers have 16Gb Ram each, the 2 CPU server has 4Gb of Ram, one of the 1 CPU servers has 6Gb Ram and the other has 2 GB of Ram.

The mixed server configuration gives us extra testing possibilities during our Proof of Concept.

For production systems we advice to use the same server specs for all cluster nodes.

1 HP-DL385 Proliant Server with 2 AMD Opteron 2,6GHz CPUs, with 2 * 73Gb local storage.

1 HP MSA 1500 Storage Array (8 Disks of 73Gb).

1 HP MSA Controller Shelf.

1 HP Storage Works SAN Switch (16 Ports, 2 Gigabit Fiber channels are used).

1 HP Procurve Switch 2824 (With 20 LAN Ports).

1 HP Storage Rack with additional Power Units, and console.

In all 5 Itanium Servers an MSA Model B Fiber Channel I/O Module is built in by HP.

Software Inventory:
The software stack used for the implementation of our Proof of Concept is the following:
Oracle 10g Release 2 Enterprise Edition 10.2.0.1.0 : for Itanium
Oracle 10g Release 2 Enterprise Edition 10.2.0.1.0 : for x86_64 (AMD).
Oracle 10g Release 2 Clusterware 10.2.0.1.0 : for Itanium

From http://otn.oracle.com the following files were downloaded:
10201_database_linuxitanium.cpio, 10201_clusterware_linuxitanium.cpio and
for the AMD Load Generator Server 10201_database_linux_x86_64.cpio.

Oracle Cluster File System Release 2 (OCFS2) : for Itanium
From the http://oss.oracle.com the following OCFS2 rpms were downloaded:
ocfs2-2.6.9-22.0.1.EL-1.0.9-1.ia64.rpm, ocfs2console-1.0.3-1.ia64.rpm
and ocfs2-tools-1.0.3-1.ia64.rpm.

Red Hat Enterprise Linux 4 Advanced Server Update 2 + Patch 0.1
HP installed RHEL4 U2 from the RHEL4 U2 Installation CDs. This version was patched with kernel-2.6.9-22.0.1.EL.ia64.rpm to match the OCFS2 kernel version.
(uname -a shows: Kernel 2.6.9-22.0.1. EL #1 SMP on the Itanium servers.

For the AMD Load Generator server HP installed RHEL4 U2 from x86_64 installation CDs.
(uname -a shows: Kernel 2.6.9-22.ELsmp #1 SMP on the AMD server).

Other Handy Tools:
To log on to the Linux systems with Secure Shell (ssh) we used Putty (0.53b).
This tool can be downloaded from:
http://www.chiark.greenend.org.uk/~sgtatham/putty/download.html

To display GUI output of the Oracle Universal Installer (OUI) we used VNC-Viewer.
Get the free VNC Viewer at www.realvnc.com .

Round Up
Now we have got enough hardware and software to build a 4 Node RAC Cluster and a Physical Standby Database.

See our next PoC Blog article: Preparation Steps on bloggingaboutoracle.org