I

Introduction

Copyright 2007, Oracle. All rights reserved.

Overview

This course is designed for anyone interested in

implementing a Real Application Clusters (RAC)
database.
Although coverage is general, most of the examples
and labs in this course are Linux based.
Knowledge of and experience with Oracle Database 10g
architecture are assumed.
Lecture material is supplemented with hands-on
practices.

I-2

Copyright 2007, Oracle. All rights reserved.

Course Objectives

In this course, you:

Learn the principal concepts of RAC
Install the RAC components
Administer database instances in a RAC and ASM
environment
Manage services
Back up and recover RAC databases
Monitor and tune performance of a RAC database
Administer Oracle Clusterware

I-3

Copyright 2007, Oracle. All rights reserved.

Typical Schedule

Topics

Lessons

Day

Introduction and installation

I, 1, 2, 3

RAC administration and tuning

4, 5, 6

7, 8

9, 10

11, 12

Advanced topics

Workshop: Cloning

I-4

Copyright 2007, Oracle. All rights reserved.

A History of Innovation
Automatic
Workload
Management
Automatic
Storage
Management

Enterprise

Grids

Grid
Control

RAC
Data
Guard
Nonblocking
queries

Low-cost
commodity
clusters

Resource
manager

OPS
I-5

Copyright 2007, Oracle. All rights reserved.

Oracle
Clusterware

What Is a Cluster?

Interconnected nodes
act as a single server.
Cluster software
hides the structure.
Public
Disks are available
network
for read and
write by all nodes.
Operating system
is the same on each Clusterware
machine.
on each node

Private Interconnect
Node
Public
network

Public
network

Disks

I-6

Copyright 2007, Oracle. All rights reserved.

Public
network

Oracle Real Application Clusters

Multiple instances
accessing the same
database
One instance
per node
Physical or
logical access
to each
database file
Software-controlled
data access

Interconnect

Shared
cache

Instances
spread
across nodes
Database
files

I-7

Copyright 2007, Oracle. All rights reserved.

Benefits of Using RAC

High availability: Surviving node and instance failures

Scalability: Adding more nodes as you need them in the
future
Pay as you grow: Paying for only what you need today
Key grid computing features:
Growth and shrinkage on demand
Single-button addition of servers
Automatic workload management for services

I-8

Copyright 2007, Oracle. All rights reserved.

Clusters and Scalability

SMP model

RAC model

Shared
storage

Memory

Cache

Cache

SGA

SGA

CPU CPU

CPU CPU

BGP BGP

BGP BGP

Cache coherency

Cache fusion
BGP (background process)

I-9

Copyright 2007, Oracle. All rights reserved.

Levels of Scalability

Hardware: Disk input/output (I/O)

Internode communication: High bandwidth and low
latency
Operating system: Number of CPUs
Database management system: Synchronization
Application: Design

I - 10

Copyright 2007, Oracle. All rights reserved.

Scaleup and Speedup

Original system
Hardware Time

Cluster system scaleup

Hardware Time
Hardware

Hardware

I - 11

Time

Up to
200%
of
task

100% of task

Cluster system speedup

Up to
300%
of
task

Hardware

Hardware

Time

Copyright 2007, Oracle. All rights reserved.

100%
of task
Time/2

Speedup/Scaleup and Workloads

I - 12

Workload

Speedup

Scaleup

OLTP and Internet

No

Yes

DSS with parallel query

Yes

Yes

Batch (mixed)

Possible

Yes

Copyright 2007, Oracle. All rights reserved.

I/O Throughput Balanced: Example

Each switch needs to support 800 MB/s

to guarantee a total system throughput
of 1600 MB/s.

FC-switch

Disk
array 1

I - 13

Disk
array 2

Disk
array 3

Each machine has 2 HBAs:

8 200 MB/s = 1600 MB/s

HBA1
HBA2

HBA1
HBA2

HBA1
HBA2

HBA1
HBA2

Each machine has 2 CPUs:

2 200 MB/s 4 = 1600 MB/s

Disk
array 4

Disk
array 5

Disk
array 6

Disk
array 7

Copyright 2007, Oracle. All rights reserved.

Disk
array 8

Each disk array

has one 2-Gbit
controller:
8 200 MB/s =
1600 MB/s

Performance of Typical Components

Throughput Performance
Component Theory (Bit/s) Maximal Byte/s
HBA Gbit/s 100/200 Mbytes/s
16 Port Switch 8 2 Gbit/s 1600 Mbytes/s
Fibre Channel 2 Gbit/s 200 Mbytes/s
Disk Controller 2 Gbit/s 200 Mbytes/s
GigE NIC

1 Gbit/s 80 Mbytes/s

Infiniband 10 Gbit/s 890 Mbytes/s

CPU 200250 MB/s

I - 14

Copyright 2007, Oracle. All rights reserved.

Complete Integrated Clusterware

9i RAC

10g Oracle Clusterware

Applications

Membership

I - 15

Cluster control/Recovery APIs

Automatic Storage Management
Messaging and locking
Membership

Connectivity

Connectivity

Hardware/OS kernel

Hardware/OS kernel

Copyright 2007, Oracle. All rights reserved.

Management APIs

Messaging and locking

Services framework
Event Services

Volume Manager
file system

System Management

Event Services

Cluster control

Applications/RAC

Necessity of Global Resources

SGA1

SGA2

SGA1

SGA2

1008

1008

1008

SGA1

SGA2

SGA1

1009

1008

1009

Lost
updates!

1008

SGA2

1008
4

I - 16

Copyright 2007, Oracle. All rights reserved.

Global Resources Coordination

Cluster

Node1
Instance1
GRD Master

LMON
LMD0
LMSx
LCK0
DIAG

Noden
Instancen

Cache
GES
GCS

Global
resources
Interconnect

GRD Master

GES
GCS

Cache

LMON
LMD0
LMSx
LCK0
DIAG

Global Resource Directory (GRD)

Global Cache Services (GCS)

I - 17

Global Enqueue Services (GES)

Copyright 2007, Oracle. All rights reserved.

Global Cache Coordination: Example

Cluster

Node1
Instance1
Cache

LMON
LMD0
LMSx
LCK0
DIAG
2
Block mastered
by instance 1

Node2
Instance2

1009

1009 Cache

Instance 2 has
the current version of the block.

GCS

1008

I - 18

Copyright 2007, Oracle. All rights reserved.

LMON
LMD0
LMSx
LCK0
DIAG
1
Which instance
masters the block?

No disk I/O

Write to Disk Coordination: Example

Cluster

Node1
Instance1
Cache

Node2
Instance2

1009

LMON
LMD0
LMSx
LCK0
DIAG

1
Need to make room
in my cache.
Who has the current version
of that block?

1010 Cache

LMON
LMD0
LMSx
LCK0
DIAG

Block flushed, make room

GCS

Instance 2 owns it.

Instance 2, flush the block
to disk.

1010

I - 19

Copyright 2007, Oracle. All rights reserved.

Only one
disk I/O

Dynamic Reconfiguration
Reconfiguration remastering
Node1

Node2

Instance1

Instance2

Instance3

masters granted

masters granted

masters granted

R1
R2

1, 2, 3
1, 3

Node1

R3
R4

2, 3
1, 2

Node2

R5
R6

2
1, 2, 3

Node3

Instance1

Instance2

Instance3

masters granted

masters granted

masters granted

R1
R2
R3

I - 20

Node3

1, 3
1, 3
3

R3
R4

2, 3
1, 2

R5
R6
R4

1, 3
1

Copyright 2007, Oracle. All rights reserved.

Object Affinity and Dynamic Remastering

Messages are sent to remote node when reading into cache.
Node2

Node1
Instance1

GCS message to master

Before
dynamic
remastering

Instance2

Read from
disk

Object

Node2

After
dynamic
remastering

Instance1
Instance2

Node1

No messages are sent to remote node when reading into cache.

I - 21

Copyright 2007, Oracle. All rights reserved.

Global Dynamic Performance Views

Retrieve information about all started instances

Have one global view for each local view
Use one parallel slave on each instance

Cluster
Node1
Instance1

GV$INSTANCE

V$INSTANCE

I - 22

Noden
Instancen

V$INSTANCE

Copyright 2007, Oracle. All rights reserved.

Additional Memory Requirement for RAC

Heuristics for scalability cases:

15% more shared pool
10% more buffer cache

Smaller buffer cache per instance in the case of

single-instance workload distributed across multiple
instances
Current values:
SELECT resource_name,
current_utilization,max_utilization
FROM
v$resource_limit
WHERE resource_name like 'g%s_%';
SELECT * FROM v$sgastat
WHERE name like 'g_s%' or name like 'KCL%';
I - 23

Copyright 2007, Oracle. All rights reserved.

Efficient Internode Row-Level Locking

UPDATE

UPDATE

Node1

Node2

Node1

Node2

Instance1

Instance2

Instance1

Instance2

No block-level
lock

COMMIT

UPDATE

Node1

Node2

Node1

Node2

Instance1

Instance2

Instance1

Instance2
3

I - 24

Copyright 2007, Oracle. All rights reserved.

Parallel Execution with RAC

Execution slaves have node affinity with the execution

coordinator but will expand if needed.

Node 1

Node 2

Node 3

Node 4

Execution
coordinator

Shared disks

I - 25

Copyright 2007, Oracle. All rights reserved.

Parallel
execution
server

RAC Software Principles

Cluster

Node1
Instance1

Instancen

Cache

Cache

Global
resources

LMON
LMD0
LMSx
LCK0
DIAG

Oracle Clusterware
CRSD & RACGIMON
EVMD

Cluster
interface

OCSSD & OPROCD

Applications
ASM, DB, Services, OCR
VIP, ONS, EMD, Listener

I - 26

Noden

LMON
LMD0
LMSx
LCK0
DIAG

Oracle Clusterware
CRSD & RACGIMON
EVMD
OCSSD & OPROCD

Global
management:
SRVCTL, DBCA, EM

Applications
ASM, DB, Services, OCR
VIP, ONS, EMD, Listener

Copyright 2007, Oracle. All rights reserved.

RAC Software Storage Principles

Node1

Noden

Instance1

Node1

Noden

Instancen

Instance1

CRS_HOME
ORACLE_HOME
ASM_HOME

CRS_HOME
ORACLE_HOME
ASM_HOME

CRS_HOME

CRS_HOME

Local storage

Local storage

Local storage

Local storage

Instancen

Voting files

Voting files

OCR files

OCR files

Shared storage

ORACLE_HOME
ASM_HOME
Shared storage

Permits rolling patch upgrades

Software is not a single
point of failure.
I - 27

Copyright 2007, Oracle. All rights reserved.

RAC Database Storage Principles

Node1
Instance1

Archived
log files

Archived
log files
Local storage

Undo tablespace
files for
instance1
Online
redo log files
for instance1

Local storage

Data files
Temp files
Control files
Flash recovery area files
Change tracking file
SPFILE
TDE Wallet
Shared storage

I - 28

Noden
Instancen

Copyright 2007, Oracle. All rights reserved.

Undo tablespace
files for
instancen
Online
redo log files
for instancen

RAC and Shared Storage Technologies

Storage is a critical component of grids:

Sharing storage is fundamental.
New technology trends

Supported shared storage for Oracle grids:

Network Attached Storage
Storage Area Network

Supported file storage for Oracle grids:

Raw volumes
Cluster file system
ASM

I - 29

Copyright 2007, Oracle. All rights reserved.

Oracle Cluster File System

Is a shared disk cluster file system for Linux and

Windows
Improves management of data for RAC by eliminating
the need to manage raw devices
Provides open solution on the operating system side
Can be downloaded from OTN:
http://oss.oracle.com/projects/ocfs/ (Linux)
http://www.oracle.com/technology/software/products/database/o
racle10g/index.html (Windows)

I - 31

Copyright 2007, Oracle. All rights reserved.

Automatic Storage Management

Provides the first portable and high-performance

database file system
Manages Oracle database files
Application
Contains data spread across
disks to balance load
Database
Provides integrated mirroring
File
across disks
system
ASM
Solves many storage
Volume
management challenges
manager
Operating system

I - 32

Copyright 2007, Oracle. All rights reserved.

CFS or Raw?

Using CFS:

Simpler management
Use of OMF with RAC
Single Oracle software installation
Autoextend

Using raw:

I - 33

Performance
Use when CFS not available
Cannot be used for archivelog files
ASM eases work

Copyright 2007, Oracle. All rights reserved.

Typical Cluster Stack with RAC

Servers
Interconnect
High-speed Interconnect: Gigabit Ethernet
UDP
Oracle Clusterware
RAC
Linux, UNIX,
Windows

RAC
Linux
Windows

RAC
Linux
Windows

ASM

OCFS

RAW

RAC
AIX, HP-UX, Solaris
ASM

Database shared storage

I - 34

Proprietary
Proprietary
OS C/W

Copyright 2007, Oracle. All rights reserved.

RAW CFS
OS CVM

RAC Certification Matrix

1.
2.
3.
4.
5.

I - 35

Connect and log in to http://metalink.oracle.com.

Click the Certify tab on the menu frame.
Click the View Certifications by Product link.
Select Real Application Clusters. Then click Submit.
Select the correct platform and click Submit.

Copyright 2007, Oracle. All rights reserved.

RAC and Services

Application server
Run-time connection load balancing
Service location transparency
Service connections

Listeners
Connection load balancing

Service availability aware

RAC instances
ERP

ERP

CRM

CRM

Backup
Priority
Alerts
Tuning

ERP

ERP

CRM

CRM

Oracle Clusterware
Up/down/LBA events notification engine
Restart failed components

I - 36

CRM

Copyright 2007, Oracle. All rights reserved.

Modify service to instance mapping.

Stop or start service connections.

ERP

Available Demonstrations

RAC scalability and transaction throughput

RAC speedup and parallel queries
Use TAF with SELECT statements

http://www.oracle.com/technology/obe/demos/admin/demos.html

I - 37

Copyright 2007, Oracle. All rights reserved.