Cluster Interconnect in Oracle RAC

In this Document

Applies to:

Purpose

The purpose of this note is to make DBAs and System Administrators familiar with the concept of the cluster or private interconnect and its usage in Oracle Clusterware and Oracle Real Application Clusters

Scope

Details

The term 'Private Interconnect' or 'Cluster Interconnect' is the private communication link between cluster nodes.
 For Oracle Real Application Clusters and Oracle Clusterware we can differentiate between
1. Physical interface (including NIC, cables and switches)
2. Private Interconnect used by Oracle Clusterware
3. Private Interconnect used by Oracle Real Application Clusters (RAC)

Why Do We Need a Private Interconnect ?

Clusterware uses the interconnect for cluster synchronization (network heartbeat) and daemon communication between the the clustered nodes. This communication is based on the TCP protocol.
RAC uses the interconnect for cache fusion (UDP) and inter-process communication (TCP). Cache Fusion is the remote memory mapping of Oracle buffers, shared between the caches of participating nodes in the cluster. The volume and traffic patterns of this type of data, shared between nodes can vary greatly depending on the applications.
There are some vendor specific protocol exceptions in Oracle 10g.

Interconnect High Availability

It is an Oracle Best Practice to make the private interconnect highly available. Depending on the operating system and vendor you can use OS network drivers such as bonding, teaming, IPMP, Etherchannel, APA, MultiPrivNIC. For that purpose the interconnect should be configured across two NICs as well as two switches for complete redundancy so that the cluster can survive a single point of failure.

The setup of the highly available interconnect is transparent to Oracle Clusterware and RAC. This means that an underlying failure is handled by the OS or networking drivers that manage the interfaces. Oracle software does not recognize the change underneath, because the failover is handled by the operating system transparently to Oracle.

Private Interconnect for RAC

The database uses the private interconnect for communication and cache fusion. The private interconnect is also referred to as "The Cluster Interconnect". Oracle highly recommends that the database and Oracle Clusterware share the same interconnect.
The instances get the private interconnect definition from

• spfile or init.ora when the parameter CLUSTER_INTERCONNECTS was set
• Oracle Cluster Regsitry (OCR) defined during the installation (default)
• automatically from platform defaults if not defined in the spfile/init.ora and OCR.

A value in the spfile or init.ora overrides a definition in the OCR.

Identification of the Private Interconnect for RAC

The value of the private interconnect for an instance can be identified using

1. The views V$CLUSTER_INTERCONNECTS and V$CONFIGURED_INTERCONNECTS
   
   V$CLUSTER_INTERCONNECTS displays one or more interconnects that are being used for cluster communication.
   
   V$CONFIGURED_INTERCONNECTS displays all the interconnects that Oracle is aware of. This view aims to answer the question on where Oracle found the information about a specific interconnect.
   
   These views are not available in Oracle 10g 10.1.
2. The alert.log
   • CLUSTER_INTERCONNECTS parameter is set
   • Value read from Cluster Registry

     Interface type 1 Database Team 10.x.x.0 configured from OCR for use as a cluster interconnect

   • Interconnect finally used

     Cluster communication is configured to use the following interface(s) for this instance
     10.x.x.1

     
     Note that there might be a difference to the interconnect defined in the OCR because the value was overridden by setting the CLUSTER_INTERCONNECTS parameter or when Oracle selected another interconnect because the one in the OCR was not available.
   • Protocol
     Unix and Linux

     cluster interconnect IPC version:Oracle UDP/IP (generic)

     
     Windows:

     cluster interconnect IPC version:Oracle 9i Winsock2 TCP/IP IPC

Private Interconnect for ASM

Oracle ASM must use the same interconnect as Oracle Clusterware.

CLUSTER_INTERCONNECTS Parameter

This parameter overrides the value configured in the OCR. You may specify multiple IP addresses in CLUSTER_INTERCONNECTS which will cause all defined interfaces to be used. Keep in mind that a failure of one of those interfaces will cause the instance to fail. The failed instance can only be restarted when all the interfaces are fixed or you start all the instances without the faulty interface.

 

References