Recovery and Backup Recovery and Backup

RMAN
TIER 1 Experience, status and questions.
Carlos Fernando Gamboa, BNL
Gordon L Brown, RAL
Meeting at CNAF
J 12 13 f 2007 B l It l
1
June 12-13 of 2007, Bologna, Italy
Table of Content
Factors that define a backup policy
B k t t (f i DBA SARA k h ) Backup strategy (from previous DBA SARA workshop)
Site status RMAN configuration
RMAN implementation problems RMAN implementation problems
Experience
Questions Questions
Example (Presented by Gordon L Brown)
3D LCG Database Administration workshop at CNAF - June 12-13, 2007 2
Factors that define a backup p
policy
Maximum % of data that can be lost
- Retention Policy: How long are the backups retained and how Retention Policy: How long are the backups retained and how
far is acceptable to recover the database?
- How many copies per file must be retained?
Time required to recover the service
- Faster recovery through disk media but more expensive y g p
than tape.
- Minimum % downtime of the service during recovery
Cost/Benefit of backup policy and recovery strategy
Site resources
3D LCG Database Administration workshop at CNAF - June 12-13, 2007 3
CERN backup strategy
RMAN - backups to tape
Level 0, every 2 weeks
level 1 cumulative twice per week level 1 cumulative, twice per week
level 1 differential, every day (except when the cumulative
backup is done)
archivelogs backups, every 30 minutes archivelogs backups, every 30 minutes
Retention: recovery window of 31 days (may change)
RMAN - backups to disk: RMAN backups to disk:
Daily refreshed with incremental recovery
Image copy delayed from production (2 days)
Allows for very fast recovery for many failure scenarios Allows for very fast recovery for many failure scenarios
Regular tests of recovery procedure
Open point: disaster recovery / dataguard
3D LCG Database Administration workshop at CNAF - June 12-13, 2007 4
From: Gordon D. Brown e-Science, RAL3D Workshop, SARA, Netherlands 20/21 March 2007
Summary Sites RMAN implementation y
Site Current Status Incremental
Level 0
Incremental level 1
cumulative
Incremental level1
differential
Archivelogs
BNL -Media:Disk
-Waiting for disk storage
equipment (IBM DS3400 3.6TB )
-Evaluating Tape resources
Retention Polic 7 da s
Weekly 2 days/week 3 days/week
(planned)
Hourly
-Retention Policy 7 days
PIC -Media:Disk
-Waiting DBA workshops input
before deploying
Bi weekly
(Planned)
Daily
(Planned)
Every 30
minutes
(Planned) before deploying (Planned)
FTS already deployed
Weekly 1 day/week
(Thursday)
2 days/week
(Friday and Saturday)
-Full
Saturday
-Hourly
ASGC F ll B k kl I t l N t d fi ASGC Full Backup weekly Incremental
(after may)
Not define
time
NDGF -Media: Disk
-Full backup
Daily Every hour
CNAF -Media: Disk
-Monthly full backup (some cases)
-Catalog database which they
stored the script executed by the
target databases
Weekly
- kept for 3
weeks
Daily
-kept for 3
weeks
-Kept for 3
days
3D LCG Database Administration workshop at CNAF - June 12-13, 2007 5
target databases.
Summary sites RMAN implementation
Site Current Status Incremental
Level 0
Incremental
Level 1
cumulative
Incremental
Level1
differential
Archivelogs
GRIDKA -Catalog in control file with keep time 31
-Control scripts on RMAN logs to monitor backup
- Installation of own GridControl on a separate OEM-
Server
-Schedule all RMAN Backups from this Server
-Weekly -Daily -Daily -Daily?
p
-Use of Tivoli Data Protector for Oracle to write directly
to tape is not fully discussed.
-Use of Nagios for monitoring backup
TRIUMF -Backup to disk: Planning to migrate to tape on late
summer
-Weekly -Planned using -Planned
summer
-The backup server is auto mounted
on the nodes of the RAC.
-NFS daemon configured to avoid concurrent backups
of other system when RMAN is backing up.
RAL U t l it f t f th W kl D il D il RAL - Use a central repository for storage of the
backup/restore control data
-Backups are validated daily. Team is notified if the
validation of the backup fails on a daily report.
-Backups are kept on disk for two weeks and the oldest
-Weekly -Daily -Daily
p p
is replaced after the new backup is successfully
validated.
-Daily secure of backups are copied to tape.
RMAN jobs run as CRON jobs "daily report" job as a
3D LCG Database Administration workshop at CNAF - June 12-13, 2007 6
-RMAN jobs run as CRON jobs daily report job as a
scheduled job in Grid Control.
RMAN
Implementation Problems
Control file block problem
An archivelog file was not found on the control file.
Different archivelog format on each node of the RAC.
Service interruption to change archivelog destination Service interruption to change archivelog destination.
3D LCG Database Administration workshop at CNAF - June 12-13, 2007 7
Sites experience with RMAN
RMAN deployment on FTS.
RMAN basic configuration through OEM.
M t f th it lid ti RMAN i t f th 3D Most of the sites are validating RMAN scripts for the 3D
database backup and learning how to use this tool.
TRIUNF experience with compression
- It seems too heavy on the oracle RAC node, and does not
t ll ffi i tl seem to compress really efficiently.
- Running bzip2 on a 1.9 GB compressed RMAN file results
in a file that is further compressed to 1 6 GB
3D LCG Database Administration workshop at CNAF - June 12-13, 2007 8
in a file that is further compressed to 1.6 GB.
Sites Experience with RMAN p
(cont.)
Backup process
Testing is tedious; with our currently-configured g ; y g
backup server a full backup takes almost 2 hours.
CNAF has experienced restore with single instance with
local file system (ext3) and with ASM for the recovery
area in remote storage system area in remote storage system.
3D LCG Database Administration workshop at CNAF - June 12-13, 2007 9
Questions Questions
Integrate RMAN / TIVOLI
Cost / benefit ratio?
OEM
- Cron jobs to implement backups or configure them by using OEM?
- How to configure OEM to send alerts when service is down?
RMAN implemented as cron jobs RMAN implemented as cron jobs
- Can cron jobs use ASM disk as backup destination when using RMAN
scripts?
Is it necessary to implement RMAN scripts on each node to archivelog - Is it necessary to implement RMAN scripts on each node to archivelog
and delete them?
3D LCG Database Administration workshop at CNAF - June 12-13, 2007 10
Questions (cont.)
How important is to implement an independent RMAN catalog
Db? Db?
Is there any reason to limit the size of an RMAN-generated file
for disk-based backup?
Are there any backup options that we should use that help to
optimize RMAN restores?
What is the recommended number of backups that sites retain? What is the recommended number of backups that sites retain?
Has someone experience to run grid control as a VMWare Virtual
Machine?
Examples for use dmbs scheduler to manage RMAN jobs node Examples for use dmbs_scheduler to manage RMAN jobs node-
independent on different Oracle RAC?
3D LCG Database Administration workshop at CNAF - June 12-13, 2007 11
Thanks to the people that contributed to this
presentation presentation.
Luis Diaz
D i D t i h Denice Deatrich
Dave Wei
Federico Bonifazi
Olli Tourunen
Doris Wochele
Gordon L. Brown
3D LCG Database Administration workshop at CNAF - June 12-13, 2007 12
Gordon L. Brown