How to Minimize Brownouts in

Oracle RAC Reconfiguration

Session ID# UGF9795
Amit K Das
PayPal

Introduction: about our team

Sehmuz Bayhan Our visionary director. Executed great changes in lightning

speed.
Saibabu Devabhaktuni Our fearless leader at PayPal for at least 10 years.
http://sai-oracle.blogspot.com/
Kyle Towle Our fearless database architect at Paypal for at least 8 years.
Dong Wang Goldengate expert, speaker at multiple conferences, PayPal
DBA for going on 8 years.
John Kanagaraj Author, Oracle ACE, frequent speaker at Oracle
conferences
Sarah Brydon One of the very few Oracle Certified Masters.
Samrat Roy One of our Expert DBA.

Who Am I?
11 years in Oracle RAC Development team.
Technical lead for worlds first Exadata production golive (Apple), while at Oracle.
Currently Engineering lead/architect for World largest
Exadata OLTP system (PayPal).
Frequent presenter in Oracle Open World, IOUG,
NoCOUG.
Love fishing.

PayPals Amazing Growth and Requirements

Amazing Growth
Exponential growth in PayPal business year over year

Business is growing rapidly

New users, features, transaction
New channels: POS, Mobile, etc

Massive growth in database demand every year

Not uncommon to see database workloads grow 50-100% every
year

One of the Largest OLTP database on Oracle

Measured by Executions X Processes (concurrency)
Fast paced VLDB OLTP environment on Oracle

500+ database instances

OLTP databases commonly 10-130 TB
5-14K concurrent processes
Executions > 80K/sec, >10GB Redo/Minute

Continuously growing
High growth of PayPals business per year up to 2 X workload
increase
Tier one databases built to support 300K+ execs/sec

Agenda
1.
2.
3.
4.
5.
6.

Understand Oracle RAC and its advantages

Understand Oracle RAC reconfiguration
Measure the reconfiguration time
Oracle RAC reconfiguration vs. business SLA
Understand the workload and behavior
Minimizing the impact of Oracle RAC reconfiguration

Understand Oracle RAC and its Advantages

Why Oracle RAC?
1.
2.
3.
4.
5.
6.

Without it, we cannot scale our fast growing business.

Without it, we will not have high availability.
The largest single machine cannot be a replacement for Oracle
RAC.
Without it, we will not able to do rolling patching.
Without it, we cannot build a real cloud.
And More

Understand RAC Reconfiguration

Every Oracle Block in RAC is protected by a lock
element, when it is accessed by an instance.
Total number of master locks will be equally distributed
between all instances.
Other non mastered instance will own a copy/shadow
lock if he does not own the master lock but interest with
that block.
Open Block count can measure from v$bh
Open lock count can measure from X$ view

Understand RAC Reconfiguration

Why do we need Reconfiguration?

To protect cache consistency when a node joins or leaves

the cluster.

When it can trigger?

If an instance joins or leaves the cluster

Start or Stop recovery of ADG

Partial reconfiguration triggers:
o
o
o

In certain situations when DRM is ON

Flushing of Buffer Cache on any node in the cluster.
When redirecting locks

Understand RAC Reconfiguration

What is the effect?

While running the reconfiguration, applications will have

transient delay to access some database objects,
depending on access pattern.

Duration of reconfiguration depends on

Size of buffer cache
Heat map of objects in cache
Number of active sessions

Understand RAC Reconfiguration

Note: We have huge OLTP load and very large

buffer cache, we observe things depending our
workload and business needs .

Measure the reconfiguration time

When node leave or join, dump the ASH data to a offline
table
Create table ash_092213 as select * from v$active_session_history;
Generate report from ash_092213 with a SQL like below
col MAXTIME format a15
col SAMPLE_TIME format a26
col BLOCKING_SESSION format 99999999
col event format a25
col AVGTIME format a15
select a.sample_time,
a.BLOCKING_SESSION,b.event,a.BLOCKING_SESSION_STATUS,a.BLOCKING_HANGCHAIN_INFO, a.event,
count(*), round(max(a.TIME_WAITED)/1000,2)||' ms' maxtime,round(avg(a.TIME_WAITED)/1000,2)||' ms' avgtime
from ash_092213 a, ash_092213 b
where a.sample_time=b.sample_time and b.SESSION_ID=a.BLOCKING_SESSION
group by a.sample_time,a.BLOCKING_SESSION,a.BLOCKING_SESSION_STATUS,b.event,
a.BLOCKING_HANGCHAIN_INFO,a.event having count(*)>5 order by a.sample_time ;

Measure the reconfiguration time

Analyze the report of last SQL

SAMPLE_TIME
BLK_SESSION Blocker_EVENT
BLK_Stat Waiter_EVENT
COUNT(*) MAXTIME
AVGTIME
-------------------------- ---------------- ------------------------- ----------- - ------------------------- ---------- --------------- --------------09-JUL-13 09.38.17.416 PM 3521 ges enter server mode
VALID log file sync
1043 6605.26 ms 2889.65 ms
09-JUL-13 09.38.17.416 PM 7419 latch: enqueue hash chains
VALID latch free
8 600.26 ms
341.55 ms
09-JUL-13 09.38.17.416 PM 9162 gcs resource directory to be unfrozen VALID gc buffer busy acquire
12
0.00 ms
0 ms
09-JUL-13 09.38.17.416 PM 4442 gcs resource directory to be unfrozen VALID gc buffer busy acquire
41
0.00 ms
0 ms
09-JUL-13 09.38.17.416 PM 18612 gcs resource directory to be unfrozen VALID gc buffer busy acquire
12
0.00 ms
0 ms
.............
.............
09-JUL-13 09.38.31.448 PM 22026 gcs resource directory to be unfrozen VALID gc buffer busy acquire
26 11117.21 ms 7216.45 ms
09-JUL-13 09.38.32.882 PM 3433 latch: gcs resource hash
VALID latch: gc element
8
59.23 ms
34.64 ms
09-JUL-13 09.38.32.882 PM 3521
VALID log file sync
528
51.22 ms
19.71 ms
09-JUL-13 09.38.32.882 PM 3697
VALID gc domain validation
14 1070.05 ms 806.82 ms
09-JUL-13 09.38.32.882 PM 14559 db file sequential read
VALID read by other session
36 186.36 ms
185.71 ms
09-JUL-13 09.38.48.997 PM 3521 log file parallel write
VALID log file sync
29
14.51 ms
11.87 ms
09-JUL-13 09.38.50.015 PM 3521 log file parallel write
VALID log file sync
6
.57 ms
.45 ms

Total Time of RAC reconfiguration time is :(09-JUL-13 09.38.32.882 PM

- 09-JUL-13 09.38.17.416 PM )=15sec+

Oracle RAC reconfiguration vs. business SLA

If your application is a DSS?

most of the time reconfiguration will be invisible for the business.

If your application is a large OLTP?

define your SLA and check whether reconfiguration can meet
your expectation

Understand the workload and behavior

Discover the data access pattern.
Is the application partitioned by node?
Is the workload random read/write accessing from all nodes?
Is the workload read mostly accessing from all nodes and write
only from one node?
Is the application configured as active/passive?

Understand the workload and behavior

Measure the object distribution across all nodes.
Maintain high uptime: avoid stopping or starting any instance
frequently (i.e. avoid reconfiguration)

How to minimize the impact of the reconfiguration in

Oracle RAC.

1. Proper sizing of buffer_cache.

2. Optimal number of LMS processes
3. Move application traffic and wait as long as
possible to cool down the buffer cache from the
node which is to be brought down.
4. Increase the bandwidth and low latency for
private interconnect.

Proper sizing of Buffer Cache

Larger Buffer cache => more objects => more time
required for RAC reconfiguration
If your application meets its I/O SLA with smaller Buffer
Cache, keep it as small as possible
Prior to increasing buffer cache size, consider effect on
RAC reconfiguration
Note: RAC will require a larger buffer cache than a single
node to serve same amount of traffic for failover situation

Increasing LMS processes optimally

Do not simply increase the number of LMS processes
Too many LMS processes cause overhead for synchronization
Need to limit the # of LMS processes when you have multiple
DBs on same cluster

Check:
The usage of interconnect
CPU usage

increase the number of LMS processes only if you have

enough bandwidth on the Interconnect and idle CPU cycles
Improve LMS performance further by applying Fix 13843646 : LONG
LATCH WAIT ON GCS RESOURCE VALIDATE LIST ONLARGE NUMBER OF
LMSES

Planning for reconfiguration

Move your traffic to other nodes for planned shutdown
Allow more time for auto remastering
Fix for bug 10415371 will help faster reconfiguration (4050% improvement) (needs latest JAN PSU with 11203)
Avoid unwanted double reconfiguration with fix 13812526
SQL>Alter system set db_cache_size=256M
scope=memory sid=<sid_name_which_is_going_down>;

From our experience

Need good amount of patient to execute special tool
perfectly.
Make sure one node can handle the entire traffic.
Transfer all connections to one node.
Aggressive lock conversion can cause more session
spike and make things worse.
Monitor the lock conversion.

Sharing the result from active node

Active Node
Before redirecting
locks

DLM Locks
18,143,660

3 Hours after

1,173,860

5 Hours after

638,420

* Note: This is from a test system Your mileage may vary!

2

Sharing the result from passive node

Passive Node

DLM Locks

Before
redirecting locks

12,340,238

3 Hours after

2,001,760

5 Hours after

435,628

Determining the best time to shutdown passive instance

Shutdown the passive instance when you see
Small number of DLM locks on passive node
Shrink the buffer_cache of passive node
Shutdown passive instance with immediate option will
cause close to ZERO impact.

Lessons learned
Working with Oracle for further improvement WIP
Do not flush buffer cache on passive node to close the open DLM
locks
But shrink buffer cache before shutdown
Optimization of LMS process count will improve reconfiguration.
The fix of bug 10415371 will help for faster reconfiguration (40-50%
improvement).
Avoid unwanted double reconfiguration with fix 13812526
Instance startup from mount to open will be impacted (2-10 sec) with
CF Enqueue contention, received the fix.- Bug 17237521 : DELAY IN
ARC0 STARTING UP

Disabled DRM using the parameter _gc_policy_time=0

My Next Presentation
Session ID: CON2468
Session Title: Tech System Refresh: Zero Downtime
with Oracle RAC/Oracle Automatic Storage
Management
Venue / Room: Moscone South - 300
Date and Time: 9/26/13, 15:30 - 16:30

COLLABORATE14.IOUG.ORG
April 7 11, 2014
The Venetian Resort,
Las Vegas, NV

Network with 5,000+ database and

application professionals
5 days of in-depth education built by
users, for users
Complimentary Pre-Conference
Workshops for IOUG registrants

Attend for free!

IOUGs Call for Speakers is now open
collaborate14.ioug.org/call-for-speakers

Training Close to Home

IOUG Master Class Series Returns
Coming soon to a city near you!
One Day Interactive Training Events to meet your
educational needs
Maximum Availability Architecture
with Oracles Larry Carpenter and Joe
Meeks
Communities
October 15: Milwaukee, WI
October 16: Rochester, NY
Performance and Internals with Craig Shallahamer
November 6: Chicago, IL
November 8: Atlanta, GA

Get more information at www.ioug.org/masterclass

or visit the IOUG kiosk in Moscone West, 2nd floor

Come See IOUG at the User Group Pavilion

Win a Free Registration to COLLABORATE 14!

Stop by the IOUG kiosk in the User Group Pavilion in

Moscone West, 2nd floor, to pick up a lucky poker chip and
see if you are a winner of a free registration to
COLLABORATE 14 in Las Vegas!

Complimentary eBook
Register Now
www.mhprofessional.com/dbsec
Use Code: db12c

30

Copyright 2013, Oracle and/or its affiliates. All rights reserved.

THANK YOU
Amit K Das (amidas@paypal.com)