CON8216 Holdsworth CON8216

Real-World
Performance
What the Real-World Performance Team Learns from Your
Automa8c Workload Repository Report
Andrew Holdsworth Graham Wood
Architect
Vice President
Real-World
Performance,
Real-World Performance,

Server Technologies Server Technologies
September 29, 2014 September 29, 2014
11/3/14
Copyright 2014, Oracle and/or its aliates. All rights reserved. |
Safe Harbor Statement

The following is intended to outline our general product direcOon. It is
intended for informaOon purposes only, and may not be incorporated
into any contract. It is not a commitment to deliver any material, code,
or funcOonality, and should not be relied upon in making purchasing
decisions. The development, release, and Oming of any features or
funcOonality described for Oracles products remains at the sole
discreOon of Oracle.
11/3/14
Real-World
Performance
11/3/14
Real-World Performance Root Causes
The database is not being

used as it was designed to
be used
The applica8on
architecture/code design
is sub-op8mal
There is a sub op8mal

algorithm in the database
11/3/14
To Fix Root Causes Requires

CHANGE
11/3/14
CHANGE
is scary and somebody will always
get upset
11/3/14
Are you willing to be an agent of

CHANGE ?

11/3/14
How Does Your OrganizaOon Do Performance ?

ConvenOonal
Real-World
Focus is on Good Enough or

What the Business Needs
Process Orientated/Part of
Quality Assurance
Spends most of the Ome on
Plaòrm Tuning Issues
Only changes things within
limited scope
Boaom up tuning approach
Looking for incremental gains
Focus on excellence and what

the Hardware and Socware
can do
Innovate excellent
performance and add
intellectual property to your
code
Everything is within scope
HolisOc top down approach
Focus on orders of magnitude
gains
11/3/14
Recent Results 1000X Project

Baseline:
4.3 Hours
Baseline:
2.4 Days
Baseline:
2.5 Hours
Code Changes:
4.3 Hours
Code Changes:
27 Mins
Code Changes:
2.5 Hours
Correct Usage:
29 Secs Correct Usage: 7.5 Mins
Correct Usage: 4 Secs
Bug Fixes:
12 Secs
Bug Fixes: 4.5 Mins
Bug Fixes:
0.90 Secs
Final:
12 Secs
Final:
4.5 Mins
Final:
0.90 Secs
Speed up:
1355.57
Speed up:
1002
Speed up:
1024
Yes we are the Crazy

Ones !
11/3/14
Program Agenda
1
Real-World Performance
OLTP
Batch
Real-World Performance Programs
11/3/14
AWR Architecture Analysis

More than just wait events and top SQL
Large amount of data in the AWR report

Tells us about the way that the system has been architected
and designed as well as about how it is performing
Ocen see common mistakes
Hyperlinks in slides are links to RWP video series
hap://oracle.com/goto/oll/rwp
AWR from online

system
Ready for Black

Friday?

AWR from Online system
TesOng system for Black Friday readiness
Cannot generate load expected on test system
Do you see any problems with this system scaling up from
this test?
Will we survive Black Friday ?
AWR Header
First-level bulleted text is
Calibri
28 pt
Second-level text (press tab
key) is
24 pt
Calibri is the only font used

in
the template
All bulleted text is sentence
case (capitalize rst leaer of
rst word)
Use bold, red, or both to
highlight ONLY KEY text
32 Cores available
Over processed
Sessions is 100x cores
Session count growing
Session leak
Dynamic connecOon pools
Cursors per session growing

Cursor leakage
Load Prole
~260 sessions acOve on
average
~40 on CPU
Only have 32 cores
System CPU limited
10 logons per second

In a stable system?
Session leaks
Dynamic connecOon pools
40% of user txns are

rollbacks
Coding for failure
Init.ora
Underscore parameters
Db_block_size=16384
Cursor_sharing=FORCE
Db_le_mulOblock_read_co
unt=32
Init.ora
Db_writer_processes=12
On a system that supports
asynchIO?
Open_cursors=2000
Per session limit
Implies cursor leaking
Init.ora
OpOmizer_index_cost_adj=5
0
Classic hack parameter
Processes=5500
Sessions=8320
Top events
Where is the Ome going?
Concurrency waits > 35% of

Ome
Library cache: mutex X
Latch:row cache objects
Typical of high CPU load
A symptom, not the problem
Row lock contenOon 18% of

Ome
IO with 7ms avg read Ome
CPU only 15% of DB Time
Log le sync?
Top SQL
Top statement
SELECT /*SHOP*/
YFS_ORDER_HEADER.*
FROM YFS_ORDER_HEADER
WHERE (ORDER_HEADER_KEY = :1 )
FOR UPDATE
12% of load
2 million execuOons
Average execuOon 0.1 sec
Top SQL
Top statement
SELECT /*SHOP*/
YFS_ORDER_HEADER.*
FROM YFS_ORDER_HEADER
WHERE (ORDER_HEADER_KEY = :1 )
FOR UPDATE
40% of RLWs are on that object
Top SQL
Next two statements
Call of
DBMS_APPLICATION_INFO
ApplicaOon instrumentaOon
14% of load
26M execuOons each
InstrumentaOon is a good thing
BUT
Not needed since Oracle 10g
Use parameters to OCI or Java
instead
Other SQL
Online Summary
Not looking good for Black Friday
System is CPU bound at test load levels

System seems to be leaking both cursors and sessions (and
maybe locks)
System is running far too many processes
High overhead applicaOon instrumentaOon
AWR from batch

system
Maxing out the Batch

Window?

AWR from batch system
Night Ome batch workload approaching Ome limit
AlternaOve plaòrms proposed to increase throughput
AWR Header
First-level bulleted text is
Calibri
28 pt
Second-level text (press tab
key) is
24 pt
44 Cores available
Over processed
Sessions is 30x cores
Calibri is the only font used

in
the template
All bulleted text is sentence
case (capitalize rst leaer of
rst word)
Use bold, red, or both to
highlight ONLY KEY text
Load Prole
~55 sessions acOve on
average
~10 on CPU
Only 25% of CPU resource used
Scope for signicantly more
thoughput
~5 logons per second
Init.ora
Underscore parameters
Cell_ooad_processing=FAL
SE
Init.ora
Unused pools congured
Java pool and large pool 3.5GB
each
Open_cursors=5000
Per session limit
Implies cursor leaking
Top events
Row lock contenOon

> 4.5sec average wait Ome
In batch?
Ocen applicaOon parallelism
Load balancing vs anity
9.5m IOs with 2ms avg read

Ome
CPU only 18% of DB Time
CPU usage underreported on
AIX7 when processor
mulOthreading enabled
Top SQL
PL/SQL calls with 64 and 128

execs
ApplicaOon parallel threads?
Top SQL
Top statement by
execuOons
3.2 M execuOons
0 rows returned
Large number of frequently

executed statements
All process a single row per
execute
Row by row batch processing
Top SQL
Top statement
UPDATE CH_ACCT_MAST
SET BAL_AVAILABLE = BAL_AVAILABLE
+ :B2 , CTR_UPDAT_SRLNO =
CTR_UPDAT_SRLNO + 1
WHERE COD_ACCT_NO = :B1
22% of DB Time
Not spending Ome on CPU or
disk IO
Object accounts for > 70% of

RLWs
Object is parOOoned
AaempOng to reduce
contenOon?
AWR Batch Summary

Plenty of resources on current hardware plaòrm
ContenOon caused by sub-opOmal applicaOon parallelism
Row by row processing of millions of rows giving away
orders of magnitude in performance vs set based processing
Where do we learn
more ?
11/3/14
Real-World Performance
Programs
11/3/14
Online
EducaOon
11/3/14
Online Video Series

hap://www.oracle.com/goto/oll/rwp
11/3/14
One Day Real-World Road

Show
11/3/14
Real-World Performance Tour in AssociaOon

with IOUG

Europe 2015
Feb 9th Finland

Feb 11th Benelux
Feb 13th Norway
Mar 17th Slovenia/CroaOa
Mar 19th England
Mar 23rd Denmark
Mar 25th Austria
Mar 27th Turkey
USA tbd
ASIA tbd
South America tbd
11/3/14
Real World Performance

Training
11/3/14
Real-World Performance Classroom Training

4 Day Class of Intensive Performance Training
Topics: SQL, OLTP, DW/BI

Classroom, Demos, Hands On, Test and Quizzes
Training given by Real World Performance Engineers
Designed for Architects, Developers and DBAs
4 months training in 4 days
Contact RWP and your local Oracle team to apply

11/3/14
Real-World Performance Classroom Training

Quotes
Fantas'c such a good course. Why cant all training be like this?
The demos schooled me
Content matches the real worldtotally dierent than courses oered by Oracle
University
Your understanding of Oracle real life problems, familiarity with computer science
basics and procedures, teaching excellence, and interpersonal skills all contributed to
a successful trainingVenkat Dulla, Qualcomm
Anyone out there claiming that he/she is an Oracle DBA must aPend RWP classes. We
are highly experienced team of DBAs and we thought weve seen it all un'l this class !
Real problems that we are dealing day-to-day, like connec'on storms, was turned
into training scenarios and explained so clearly with cause and solu'ons along with a
diagnos'c methodologyit was also an eye-opener for OLTP DBAs that I/O is no
longer a boPleneck with exadata and it is 'me to start using set opera'ons to
achieve order of magnitude performance gains Me'n Ylimaz, Turkcell
11/3/14
Safe Harbor Statement

The preceding is intended to outline our general product direcOon. It is
intended for informaOon purposes only, and may not be incorporated
into any contract. It is not a commitment to deliver any material, code,
or funcOonality, and should not be relied upon in making purchasing
decisions. The development, release, and Oming of any features or
funcOonality described for Oracles products remains at the sole
discreOon of Oracle.
Note: The speaker notes for this slide
include instrucOons for when to use
Safe Harbor Statement slides.
Tip! Remember to remove this text
box.

CON8216 Holdsworth CON8216

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

CON8216 Holdsworth CON8216

Uploaded by

Copyright:

Available Formats

Real-World

Safe Harbor Statement

Real-World Performance Root Causes

The database is not being

There is a sub op8mal

To Fix Root Causes Requires

Are you willing to be an agent of

How Does Your OrganizaOon Do Performance ?

Focus is on Good Enough or

Focus on excellence and what

Recent Results 1000X Project

29 Secs Correct Usage: 7.5 Mins

Correct Usage: 4 Secs

Bug Fixes: 4.5 Mins

Copyright 2014, Oracle and/or its aliates. All rights reserved. |

Yes we are the Crazy

Real-World Performance Programs

AWR Architecture Analysis

Large amount of data in the AWR report

Copyright 2014, Oracle and/or its aliates. All rights reserved. |

AWR from online

Ready for Black

Copyright 2014, Oracle and/or its aliates. All rights reserved. |

Calibri is the only font used

Cursors per session growing

Copyright 2014, Oracle and/or its aliates. All rights reserved. |

10 logons per second

40% of user txns are

Copyright 2014, Oracle and/or its aliates. All rights reserved. |

Copyright 2014, Oracle and/or its aliates. All rights reserved. |

Copyright 2014, Oracle and/or its aliates. All rights reserved. |

Copyright 2014, Oracle and/or its aliates. All rights reserved. |

Concurrency waits > 35% of

Row lock contenOon 18% of

Copyright 2014, Oracle and/or its aliates. All rights reserved. |

40% of RLWs are on that object

Copyright 2014, Oracle and/or its aliates. All rights reserved. |

Copyright 2014, Oracle and/or its aliates. All rights reserved. |

System is CPU bound at test load levels

Copyright 2014, Oracle and/or its aliates. All rights reserved. |

AWR from batch

Maxing out the Batch

Copyright 2014, Oracle and/or its aliates. All rights reserved. |

Calibri is the only font used

Copyright 2014, Oracle and/or its aliates. All rights reserved. |

~5 logons per second

Copyright 2014, Oracle and/or its aliates. All rights reserved. |

Copyright 2014, Oracle and/or its aliates. All rights reserved. |

Copyright 2014, Oracle and/or its aliates. All rights reserved. |

Row lock contenOon

9.5m IOs with 2ms avg read

PL/SQL calls with 64 and 128

Copyright 2014, Oracle and/or its aliates. All rights reserved. |

Large number of frequently

Copyright 2014, Oracle and/or its aliates. All rights reserved. |

Object accounts for > 70% of

Copyright 2014, Oracle and/or its aliates. All rights reserved. |

AWR Batch Summary

Copyright 2014, Oracle and/or its aliates. All rights reserved. |

Online Video Series

One Day Real-World Road

Real-World Performance Tour in AssociaOon

Feb 9th Finland