Cache Buffer Chain

(CBC)
Analysis

Craig A. Shallahamer - craig@orapub.com

CBC Analysis

This presentation was delivered by Craig Shallahamer

at the PHLOUG Meeting on 13-Feb-2013.

Objectives
•  Understanding CBC internals; structures and
algorithms

•  Learn how to methodically diagnose specific CBC latch

contention problems.

•  Learn three common CBC problem causes; hot index

root block, dispersed high CBC concurrency, and
singular high CBC concurrency.

•  Using Statspack, AWR, or the v$ views, learn how to

determine if CBC latch contention resolution is worth
your time and effort.

CBC Analysis
 OraPub is about Oracle performance.
•  OraPub is all about Oracle performance analysis; Resources
systematic and quantitative firefighting and
predictive analysis. •  Research Blog
•  Web site started in 1995 and the company was •  Training
founded in 1998 by Craig Shallahamer.

•  OraPub has always been about disseminating

•  Free Tools
Oracle database centric technical information. •  Free Papers
•  Consulting, training, books, papers, and products •  Books
are now being offered.

•  We have been on-site in 31 countries and our

•  Products
resources have been received in probably every •  Consulting
country where there are DBAs.

•  OraPub has its own LinkedIn Group!

CBC Analysis

This presentation was delivered by Craig Shallahamer

at the PHLOUG Meeting on 13-Feb-2013.

Short resume...kind of...

•  Studies economics, mathematics, and computer science at
university in California, US.
•  Started working with Oracle technology in 1989 as a Forms 2.3
developer on Oracle version 5.
•  Soon after started performance firefighting...daily!
•  Co-found both Oracle’s Core Technology and System
Performance Groups.
•  Left Oracle to start OraPub, Inc. in 1998.
•  Authored 24+ technical papers and worked in 26 countries.
•  Authors and teaches his classes Oracle Performance
Firefighting, Adv Oracle Performance Analysis, and Oracle
Forecasting & Predictive Analysis.
•  Authored the books, Forecasting Oracle Performance and
Oracle Performance Firefighting.
•  Oracle ACE Director.
•  Craig’s performance research blog: A Wider View

CBC Analysis
 My two books...

CBC Analysis

This presentation was delivered by Craig Shallahamer

at the PHLOUG Meeting on 13-Feb-2013.

Oracle’s three fundamental caches.

•  Database block buffer cache. With the exception
of direct block reads, before any database block can
be accessed, it must be physically read from disk and
placed into the block buffer cache.
•  Shared pool. When processing related information
needs to be shared between users (SQL, programs,
MTS, etc.) the information is usually placed into the
shared pool.
•  Redo log buffer. "Any" change made to a cached
database block is recorded into the redo log buffer.

CBC Analysis
 The Oracle Block & Buffer…
Variable Space
(internal control structures)

Free Space

Row Space

CBC Analysis

This presentation was delivered by Craig Shallahamer

at the PHLOUG Meeting on 13-Feb-2013.

A buffer can be pinned, dirty, or free.

•  Free/mirrored. When a block's cached image mirrors the actual block
on disk. The cached block (I.e., the buffer) can be replaced by another
block and it can be referenced by any Oracle process. When the
contents of a dirty buffer has been written to disk, it is once again a
mirror of the actual block and placed back on the LRU end of its LRU
list. An empty buffer is also called a free buffer. There are multiple buffer
states (x$bh.state) related to a free/mirrored buffer.
•  Pinned. A buffer is pinned to keep it from being replaced. You don’t
want a block you’re referencing to be replaced! Pinning can also be
used to ensure serial access.
•  Dirty. When a block's cached image is not the same as its database file
image. After a block has been changed (I.e., was pinned) and has not
yet been written back to its database file, it is deemed “dirty.” (v
$bh.dirty=‘Y’)

•  v$bh and sys.x$bh contain buffer details.

CBC Analysis
 The Oracle Segment

CBC Analysis

This presentation was delivered by Craig Shallahamer

at the PHLOUG Meeting on 13-Feb-2013.

The Oracle Index Segment

http://shallahamer-orapub.blogspot.com/2011/11/are-you-sure-its-index-root-block.html!
CBC Analysis
 Identifying an index root block.
SQL> select blevel, leaf_blocks, num_rows from dba_indexes where index_name = 'FINDME_C2';!
!
!
BLEVEL LEAF_BLOCKS NUM_ROWS!
---------- ----------- ----------!
2 500 100000!
!
SQL> select header_file,header_block from dba_segments where segment_name = 'FINDME_C2';!
!
HEADER_FILE HEADER_BLOCK!
----------- ------------! The root block is the
!
1 412480! header block + 1.
SQL> select dbms_utility.make_data_block_address(1,412481) from dual;!
!
DBMS_UTILITY.MAKE_DATA_BLOCK_ADDRESS(1,412481)!
----------------------------------------------!
4606785!
Remember this DBA!
!
SQL> select object_id from dba_objects where object_name = 'FINDME_C2';!
!
OBJECT_ID!
----------!
110869!
Must “treedump” by object_id.
!
SQL> alter session set events 'immediate trace name treedump level 110869';!
!
Session altered.!
!

CBC Analysis

This presentation was delivered by Craig Shallahamer

at the PHLOUG Meeting on 13-Feb-2013.

Identifying an index root block.

This is the
$ oracle@fourcore trace]$ cat prod18_ora_19918.trc! DBA of the
...!
*** 2011-09-10 11:04:51.471! header block
----- begin tree dump!
branch: 0x464b41 4606785 (0: nrow: 2, level: 2)!
+ 1.
branch: 0x464dfe 4607486 (-1: nrow: 251, level: 1)!
leaf: 0x464b42 4606786 (-1: nrow: 200 rrow: 200)!
leaf: 0x464b43 4606787 (0: nrow: 200 rrow: 200)!
leaf: 0x464b44 4606788 (1: nrow: 200 rrow: 200)!
leaf: 0x464b45 4606789 (2: nrow: 200 rrow: 200)!
...!
branch: 0x464ef7 4607735 (0: nrow: 249, level: 1)!
leaf: 0x464dfd 4607485 (-1: nrow: 200 rrow: 200)!
leaf: 0x464dff 4607487 (0: nrow: 200 rrow: 200)!
...!
leaf: 0x464ef6 4607734 (247: nrow: 200 rrow: 200)!
----- end tree dump!
$!
!

CBC Analysis
 There are three key lists...

CBCs

LRUs

WLs
CBC Analysis

This presentation was delivered by Craig Shallahamer

at the PHLOUG Meeting on 13-Feb-2013.

Visualizing 400 BHs fully connected.

400 BHs!
350 CBCs!
2 LRUs!
20% dirty!
No cloning!

CBC Analysis
 Buffer headers : the map.

CBC Analysis

This presentation was delivered by Craig Shallahamer

at the PHLOUG Meeting on 13-Feb-2013.

Visualizing 3 BHs fully connected.

CBC Analysis
 CBC Analysis

This presentation was delivered by Craig Shallahamer

at the PHLOUG Meeting on 13-Feb-2013.

Focus: cache buffer chains

CBC Analysis
 Massive memory structure...

Item 8i 9i 10g 11g

Data Buffers 76,800 699,994 327,680 32,256 (8k)

Hash Chain
153,600 1,400,000 1,048,576 65,536
Buckets

CBC Latches 1,024 8,192 4,096 2,048

CBC Buckets/
153 170 256 32
CBC Latch

CBC Analysis

This presentation was delivered by Craig Shallahamer

at the PHLOUG Meeting on 13-Feb-2013.

SQL> @rtpctx! Is there significant CBC latch contention?

Remember: This report must be run twice so both the initial and!
final values are available. If no output, press ENTER about 20 times.!
!
Database: prod23 10-OCT-12 02:15pm!
Report: rtpctx.sql OSM by OraPub, Inc. Page 1!
Response Time Activity (127 sec interval)!
!
Avg Time Time Wait!
RT Component % RT % WT Waited (ms) (sec) Count(k)!
-------------------------------------- ------- ------- ----------- ----------- --------!
CPU consumption: Oracle SP + BG procs 98.28 0.00 0.000 1291.832 0!
latch: cache buffers chains 1.35 78.45 5.631 17.760 3!
control file parallel write 0.18 10.64 33.472 2.410 0!
latch free 0.08 4.81 6.527 1.090 0!
log file parallel write 0.03 1.86 5.122 0.420 0!
. . .!
!
Arrival Arrival Service Queue Response
!
Rate Rate Time Time Time
!
Instance Workload Statistic Interval Work (work/sec) (work/ms) (ms/work) (ms/work) (ms/work)
!
---------------------------- --------------- ------------- ------------- ------------ ------------ ------------ !
user commits 27 0.123 0.0001 47845.61481 838.51852 48684.13333
!
user calls 77 0.352 0.0004 16777.03377 294.02597 17071.05974
!
session logical reads 248920484 1136623.215 1136.6232 0.00519 0.00009 0.00528
!
physical read total IO reque 418 1.909 0.0019 3090.50622 54.16268 3144.66890
!
physical write total IO requ 213 0.973 0.0010 6064.93709 106.29108 6171.22817
!
physical reads 360 1.644 0.0016 3588.42111 62.88889 3651.31000
!
physical read bytes 2949120 13466.301 13.4663 0.43804 0.00768 0.44572
!
physical writes 45 0.205 0.0002 28707.36889 503.11111 29210.48000
!
physical write bytes 368640 1683.288 1.6833 3.50432 0.06141 3.56573
!
parse count (total) 761 3.475 0.0035 1697.54481 29.75033 1727.29514
!
parse count (hard) 46 0.210 0.0002 28083.29565 492.17391 28575.46957
!
execute count 2696 12.311 0.0123 479.16602 8.39763 487.56365
!

CBC Analysis
 Q: Is the buffer in the cache?
A: Check a hash chain.
•  To quickly locate a buffer in the cache, Oracle uses a
hashing and linked list / hash chain combination.

•  If a buffer header is not on the appropriate hash chain,

Oracle knows it’s not in the cache and must make a IO
subsystem request for the block (PIO).

•  First, a free buffer is found (scan an LRU) and pinned,

then the block is requested and retrieved (a PIO request
and perhaps a wait occurrence), placed into the free
buffer, and the final internal adjustments are made. This
consumes CPU cycles and IO resources, which would
not occur if the buffer was already in the cache!
CBC Analysis

This presentation was delivered by Craig Shallahamer

at the PHLOUG Meeting on 13-Feb-2013.

Hashing avoids “tree walking”

and is extremely fast.
•  Hash functions take one or more input parameters and
transforms them into a hash value. The hash value is then
hashed to a numeric value (or address) within a specified
range. The hash value/address points to one and only one
hash bucket. Hash functions are math-based not search-
based, and therefore can be extremely fast. Oracle; inputs
are file number and block number.
•  Hash buckets are the head of the hash chains. To reduce
bucket collisions, increase the number of hash buckets.
•  Hash chains are sequential linked lists related to a single
hash bucket. These lists are usually searched sequentially
and therefore should be as short as possible. Hash chain
nodes can contain data or point to another structure (e.g.,
in memory or on disk). Oracle; consists of buffer headers.
CBC Analysis
 How hashing works.
Hash buckets

0
Input: 10,504
1

4
Hash Value
Creation 5

7
Hash Bucket
Determination 8

Hash chains

CBC Analysis

This presentation was delivered by Craig Shallahamer

at the PHLOUG Meeting on 13-Feb-2013.

A block hashing example.

$ cat block_hash.pl!
#!/usr/bin/perl!
. . .!
$buckets = $ARGV[0];!
$file_no = $ARGV[1];!
$blk_no = $ARGV[2];!
!
$hash_value = $file_no * 2102950191 + $blk_no * 102950723;!
!
$the_bucket = $hash_value % $buckets ;!
!
print " Simple block hashing example with bucket assignment. \n\n";!
print " Inputs\n";!
print " buckets = $buckets \n";!
print " file number = $file_no \n";!
print " block number = $blk_no \n";!
print " Outputs\n";!
print " hash value = $hash_value \n";!
print " bucket assignment = $the_bucket \n";!
print "\n";!

CBC Analysis
 A block hashing example.
$ perl hash_block.pl 10 12 398!
Simple block hashing example with bucket assignment. !
!
Inputs!
buckets = 10 !
file number = 12 !
block number = 398 !
Outputs!
hash value = 66209790046 !
bucket assignment = 6 !
!
$ perl hash_block.pl 10 12 399!
Simple block hashing example with bucket assignment. !
!
Inputs!
buckets = 10 !
file number = 12 !
block number = 399 !
Outputs!
hash value = 66312740769 !
bucket assignment = 9 !

CBC Analysis

This presentation was delivered by Craig Shallahamer

at the PHLOUG Meeting on 13-Feb-2013.

Multiple latches provide

increased granularity.
Option “A” Option “B”

0 0

1 1

2 2

3 3

4 4

5 5

6 6

7 7

8 8

9 9

CBC Analysis
 We want very short and
non popular chains!

CBC Analysis

This presentation was delivered by Craig Shallahamer

at the PHLOUG Meeting on 13-Feb-2013.

Popular or longs chains are

problematic.

Chains are typically popular because of

a popular buffer, but sometimes
because of a very long chain.

CBC Analysis
 CBC latch acquisition patterns.
•  Acquisition patterns are typically either disperse and wide-
spread (i.e., singular).
•  Disperse patterns are easy to eradicate by adding
additional CBC latches (Oracle centric solution).
•  Singular pattern eradication requires additional analysis to
determine why the latch is so active.

http://shallahamer-orapub.blogspot.com/2011/11/cbc-latch-diagnosis-acquisition.html!
CBC Analysis

This presentation was delivered by Craig Shallahamer

at the PHLOUG Meeting on 13-Feb-2013.

Determining the CBC latch

acquisition pattern.
select p1 latch_addr,! Careful... you
p2 latch_#! want lots of
from v$session!
where status = 'ACTIVE'! samples!
and wait_time = 0!
and event = 'latch: cache buffers chains'!
/!
!
LATCH_ADDR LATCH_#! Singular
---------- ----------! acquisition pattern
1993834384 150!
1993834384 150!
(same address)
1993834384 150!
1993834384 150!

CBC Analysis
 def latch=150!

More
def sleep=300!
drop table op_interim!
/!
create table op_interim as!
select addr,gets,sleeps!
scientific
from v$latch_children!
where latch# = &latch! ... more
/!
exec dbms_lock.sleep(&sleep);!
select t1.addr,!
data
t1.gets-t0.gets delta_gets,!
t1.sleeps-t0.sleeps delta_sleeps! during
problem
from op_interim t0,!
(!
select addr,gets,sleeps!
from v$latch_children!
where latch# = &latch!
times.
) t1!
where t0.addr = t1.addr!
order by 3,2!
/! script: latchchild.sql (OSM)
!

CBC Analysis

This presentation was delivered by Craig Shallahamer

at the PHLOUG Meeting on 13-Feb-2013.

Disperse or Singular?
ADDR DELTA_GETS DELTA_SLEEPS!
-------- ------------ -------------!
ADDR DELTA_GETS DELTA_SLEEPS!
...! -------- ------------- ------------!
76D1ABC4 302,356 5! ...!
76D23210 309,631 5! 76C33818 198,038 3!
76C8413C 324,284 5! 76C644A8 81,535 4!
76C75428 232,780 6! 76C7759C 96,993 4!
76CFCFAC 247,324 6! 76C2D1CC 145,096 4!
76E03FEC 149,355 4!
76DB79B0 247,332 6!
76CDD394 160,718 4!
76BFE7A8 253,808 6! 76C68904 182,300 4!
76DB9C98 281,330 6! 76D69374 65,250 5!
76C534AC 323,395 6! 76D7800C 133,134 5!
76D0BCC0 337,938 6! 76DA2650 156,578 5!
76C17DF0 203,694 7! 76C02B88 158,293 5!
76C04DF4 240,050 7! 76CE59E0 145,169 7!
76C7B9F8 158,243 7!
76DE64CC 295,872 7!
76CFF120 65,228 9!
76BF5EF0 232,782 8! 76CFCF30 148,187 11!
76D05864 274,886 8! 76D38668 966,345 62!
! 76CDF508 981,384 96!
1024 rows selected.! 76DAF26C 1962,752 187!
!
1024 rows selected.!

CBC Analysis
Dispersed: More latches can reduce Rt.

http://shallahamer-orapub.blogspot.com/2011/07/cbc-latches-cpu-consumption-and-wait.html!

CBC Analysis

This presentation was delivered by Craig Shallahamer

at the PHLOUG Meeting on 13-Feb-2013.

Seeing the difference

from a UOWTBA perspective.

250!

512!
1024!

CBC Analysis
Two common “singular” situations.
•  When there is a very active CBC latch, if the
concurrency gets high enough, eventually you'll see
CBC latch contention. Two common situations are:
–  A single block contains reference information but it's not
being changed often. If the issue was lots of change
occurring, we'd likely see a buffer busy wait.
–  A popular index's root block. Any time an index is
accessed its single root block must first be accessed,
which repeatedly forces the question, "Is this index root
block in the buffer cache?”.
•  The diagnostic path is to determine which buffers are
covered by the busy CBC latch and then which
buffers are the busiest within that single latch.
CBC Analysis

This presentation was delivered by Craig Shallahamer

at the PHLOUG Meeting on 13-Feb-2013.

Buffers related to busy CBC latches.

drop table op_interim;!
create table op_interim (hladdr raw(4), file# number, !
block# number, tch number);!
declare!
i number;!
begin!
for i in 1..300!
loop!
insert into op_interim !
select hladdr, file#, dbablk, tch!
from x$bh!
where hladdr in ('76DAF26C','76CDF508','76BE93CC');!
dbms_lock.sleep(1);!
end loop;!
end;!
/!
These are the most
active CBC latch
addresses.

CBC Analysis
Buffers related to busy CBC latches.
select hladdr, file#, block#,!
count(*), min(tch) min, median(tch) med,!
round(avg(tch)) avg, max(tch) max!
from op_interim!
group by hladdr, file#, block#!
order by 6!
/!
!
HLADDR FILE# BLOCK# MIN MED AVG MAX!
-------- ------ ---------- ---------- ---------- ---------- ----------!
...!
76BE93CC 1 39364 6 6 6 6!
76BE93CC 1 69730 13 14 14 14!
76CDF508 1 117329 1 36 64 181!
76DAF26C 1 117328 70 125 125 180!
76BE93CC 4 1552339 76 131 131 186!
!
42 rows selected.!

Two buffers are busiest... What type of buffers

are they and why are they so active?
CBC Analysis

This presentation was delivered by Craig Shallahamer

at the PHLOUG Meeting on 13-Feb-2013.

Checking out buffer 1,117328

SQL> l!
1 select owner, segment_name, segment_type!
2 from dba_segments!
3 where header_file=&hdr_file!
4* and header_block=&hdr_block!
SQL> /!
Enter value for hdr_file: 1!
Enter value for hdr_block: 117328!
!
OWNER SEGMENT_NAME SEGMENT_TYPE!
---------- ------------------------- ------------!
SYSTEM OP_LOAD_PARAMS TABLE!

This table contains load

intensity control information.
CBC Analysis
 Checking out buffer 4,1552339
SQL>@objfb 4 1552339!
. . . !
File number :4!
Block number :1552339!
Owner
Segment name
:MG!
:SPECIAL_CASES_BOGUS! Do you see
Segment type :INDEX!
Tablespace
File name
:USERS!
:/u01/oradata/prod18/users01.dbf!
it?
!
. . . !
SQL> l!
1 select owner, segment_name, segment_type!
2 from dba_segments!
We are
dealing with
3 where header_file=&hdr_file!
4* and header_block=&hdr_block!
!
SQL> /!
Enter value for hdr_file: 4! an index root
block!
Enter value for hdr_block: 1552338!
!
OWNER SEGMENT_NAME SEGMENT_TYPE!
---------- ------------------------- ------------!
MG SPECIAL_CASES_BOGUS INDEX!
!
1 row selected.!

CBC Analysis

This presentation was delivered by Craig Shallahamer

at the PHLOUG Meeting on 13-Feb-2013.

Long CBC chains: Oracle can

cache multiple copies of a block.
•  For read consistency and performance, Oracle commonly
keeps multiple copies of a buffer in cache.
•  These buffer “clones” are the result of buffer changes AND
processes needing a read consistent (back-in-time) buffer
view.
•  Oracle constructs an SCN time appropriate view of the
buffer, known as a CR or “stale” buffer. (v$bh.stale=‘Y’)
•  If a server process needs an “old” buffer, the CU buffer is
copied/cloned, undo is retrieved (ITL contains pointer to start
of “chain of undo”), and the undo is applied to the clone until
the appropriate SCN time has been reached. This consumes
a tremendous amount of CPU and possible IO.
CBC Analysis
 Determine hash chain length.
@clone!
. . .!
Database: prod18! ! ! ! 26-JAN-10 09:35am!
Report: clone.sql OSM by OraPub, Inc. Page 3!
Buffer Clone Summary Report!
!
File # Block # STATUS Clones!
------- ------- -------- -------!
. . . !
4 231 cr 6!
2 420 cr 6! SQL> @objfb 4 225!
4 229 cr 26! !
2 29555 cr 46! Database: prod18! ! ! ! ! 26-JAN-10 09:36am!
4 225 cr 2031! Report: objfb.sql ! OSM by OraPub, Inc. ! ! Page
! ! 1!
For file|block details run, objfb.sql <file#>! <block#>!
Object Details For A Given File #(4) and block #(225)!
SQL> l ! !
1 select file# fl,! File number :4!
2 ! block# blk,! Block number :225!
3 ! status,! Owner :OE!
4 ! count(*)-1 cnt!
Segment name :ORDERS!
5 from v$bh! Segment type :TABLE!
6 group by file#, block#, status! Tablespace :USERS!
7 having count(*) > 1! File name :/u01/oradata/prod18/users01.dbf!
8* order by cnt! !
SQL>!

CBC Analysis

This presentation was delivered by Craig Shallahamer

at the PHLOUG Meeting on 13-Feb-2013.

Increasing CBC performance.

•  Wait event. Latch: cache buffer chains
•  Solutions:
–  CPU power. The application and Oracle requirements exceed the CPU
subsystem’s capacity; faster, more, reduce other process consumption.
–  Poorly tuned SQL. This can result in an enormous number of LIOs which stress
the CBC latches and consume CPU cycles.
–  Singular child latch activity. Determine the hot buffer(s) and make them less
popular. For index root blocks, consider a partitioning strategy. For data blocks,
consider reducing the block density.
–  Disperse child latch activity. If child latch activity is dispersed, Increase the
number of hash chain latches by increasing the instance parameter
_db_block_hash_latches. This decreases the number of chains per latch.
–  Hash chain length. Determine the longest hash chain by querying, v$bh
(clone.sql). If there is one chain that is much longer than the rest, determine the
block details. Once the block details have been determined, talk with an application
person…fix the application.
–  Increase buckets by increasing instance parameter _db_block_hash_buckets.
Only works if multiple long chains.
CBC Analysis
Get the tool: OP search, “bc visual”

CBC Analysis

This presentation was delivered by Craig Shallahamer

at the PHLOUG Meeting on 13-Feb-2013.

Want to dig deeper?

•  Craig’s Performance Research Blog – A Wider View
•  Training from OraPub
Philadelphia
–  1 Day Performance Research Seminar April 22-26
–  Oracle Performance Firefighting (I)
–  Adv Oracle Performance Analysis (II)
•  Tools
–  BC Visualization Tool (www.orapub.com)

•  Books
–  Oracle Performance Firefighting (C. Shallahamer)

CBC Analysis
Thank You!

CBC Analysis

This presentation was delivered by Craig Shallahamer

at the PHLOUG Meeting on 13-Feb-2013.

Cache Buffer Chain

(CBC)
Analysis

Craig A. Shallahamer - craig@orapub.com

CBC Analysis