Using RMAN Incremental SCN to recover Archive log

standby Database

with missing

Product : Database Version Affected Environments :

RDBMS 10g and Higher only Databases with DR / DG Corrupted/unavailable archived Redo log which is required to fill an archive gap in a DR environment

Issue Description :

Issue Detail: DR/DG is in an unrecoverable state as the archived redo log which is required to fill an archive gap is unavailable due to corruption or deletion. DR/DG has to be rebuilt from scratch from the backup to fix the issue. Solution: RMAN incremental backup from SCN was used to sync a DR/DG by backing up only the changed blocks on the source from the SCN of the missed archived redo log. This backup can then be applied on the DR/DG which will update all the block changes.

STEP 1 Find SCN at Source/Primary 1. Identify the sequence# of the missing redo log file from the primary database using the below SQL.

select thread#, sequence#, first_change#, NEXT_CHANGE# from v$archived_log where (thread#, sequence#) in (select thread#, max(sequence#) seq from gv$archived_log where NEXT_CHANGE# < (select FIRST_CHANGE# from v$archived_log where sequence#=<deleted seq number> and thread#=<thread no of deleted seq no> ) group by thread# )

If any issues with this sql, use the alert log to find the closest SCN. Make sure the SCN is older than the missing archived redo log, otherwise the backup will be of no use. You can also use the below sql to find the SCN for a specific time. Eg. if the archive file which is missing got generated on 01/02/2011 02:45, find the SCN which got generated at '01/02/2011 02:30:54' or before.

col scn format 999999999999999999999 select timestamp_to_scn(to_timestamp('01/02/2011 02:30:54','DD/MM/YYYY HH24:MI:SS')) as scn from dual;
STEP 2 Run RMAN Incremental Backup at Source/Primary

1. Execute the RMAN Incremental backup at the source/primary from the SCN identified at STEP 1. rman target / run { allocate channel ch1 type disk format '<PATH>/rman_%U_bpiece1.bak'; allocate channel ch2 type disk format '<PATH>/rman_%U_bpiece2.bak'; ..... ..... BACKUP as compressed backupset INCREMENTAL FROM SCN <SCN_Value> DATABASE tag 'FORSTANDBY'; } Note: Decide the number of channels based on the CPU availability and the CPU utilization. Use the below sample Shell Script for RMAN Incremental Backup Please modify the script for setting the Environment according to the database.

#!/bin/ksh print "Starting at `date +%x_%X`" . /usr/local/bin/setdb PRODTEST2 rman target / <<EOF run { allocate channel ch1 type disk format '/rman_PRODTEST_01/rmanfiles/rman_%U_bpiece1.bak'; allocate channel ch2 type disk format '/rman_PRODTEST_01/rmanfiles/rman_%U_bpiece2.bak'; allocate channel ch3 type disk format '/rman_PRODTEST_01/rmanfiles/rman_%U_bpiece3.bak'; allocate channel ch4 type disk format '/rman_PRODTEST_01/rmanfiles/rman_%U_bpiece4.bak';

allocate channel ch5 type disk format '/rman_PRODTEST_01/rmanfiles/rman_%U_bpiece5.bak'; BACKUP as compressed backupset INCREMENTAL FROM SCN 11011575822925 DATABASE tag 'FORSTANDBY'; } exit EOF print "Completed at `date +%x_%X`"
Note: If you are not comfortable in setting the environment in script and running through script, you can run only the RMAN commands. Make sure you are running in SCREEN, since the Incremental backup will run for long time depending on the transactions happened on the Database. From Previous scenarios, for a DR/DG lag of 10 Hours the incremental backup ran for around 2 Hours in primary.
STEP 3 Create Standby controlfile at Source/Primary

Create standby controlfile at the primary/source database using the below sql SQL> alter database create standby controlfile as '/home/oracle/incrementalrman.ctl';

STEP 4 Copy backup pieces and controlfile to DR/DG

1. Copy all the backup pieces generated at the incremental backup location specified on STEP 2 (/rman_PRODTEST_01/rmanfiles) on source/primary to the DR/DG server. Note: Get help of SA to copy using multiple rsync parallely or any other faster method of copy to save time. 2. Copy the standby control file to DR/DG Server generated at STEP3

STEP 5 Pre-tasks before recovery at DR/DG

1. Cancel the current recovery at DR/DG 2. Shutdown DR/DG database

Shutdown normal;
3. Backup current standby controlfile to a backup location

dd if=<control_file_path> of=<new control_file_path>

4. Overwrite the standby controlfile with the copied standby control file created from primary.

dd if=<stndby_control_file_created_production> of=<current_location_of standby controlfile>

5. Set control_file_record_keep_time Set this parameter to make sure controlfile can hold all records of RMAN otherwise it will not be able to catalog all the backup pieces

Set in init<db>.ora > control_file_record_keep_time=7

3

6. Start the DR/DG Database

startup nomount; alter database mount standby database;

7. Create missing datafiles Create the missing datafiles that were added to production database after the <missing sequence> was generated. Note: Run the following Query in Primary for finding the Creation_time. Compare the time with the timing of missing sequence and add the datafiles in DR/DG.

select t.name "TABLESPACE_NAME",f.name "DATAFILE" ,to_char(creation_time,'DD-MON-YYYY HH24:MI:SS') "CREATION_TIME", f.bytes/1024/1024/1000 "SIZE_IN_GB" from v$datafile f,v$tablespace t where f.ts#=t.ts# order by 3
Run the Following in DR/DG after mounting the standby database :

ALTER SYSTEM SET standby_file_management='MANUAL' SCOPE=MEMORY; alter database create datafile '<datafile_file_path as in production>'
8. Catalog the incremental backup Catalog the incremental backup taken at STEP 2 to the DR/DG database.

rman target /
CATALOG START WITH '<PATH where the Files copied in Standby Server >' ; Eg,

CATALOG START WITH '/oracle/dump/PRODTEST2/rman_incr_copied' ;

STEP 6 Recover DR/DG with the Incremental backup

Run the RMAN RECOVER command with the NOREDO option to apply the incremental backup to the DR/DG database. All the changed blocks are updated at the standby to make it, up to date with the primary/source database. rman target / { allocate channel ch1 type disk; allocate channel ch2 type disk; .. RECOVER DATABASE noredo; }

Use the below sample Script for RMAN Incremental recovery

#!/bin/ksh rman target / <<EOF

4

run { allocate channel ch1 type disk; allocate channel ch2 type disk; allocate channel ch3 type disk; allocate channel ch4 type disk; allocate channel ch5 type disk; RECOVER DATABASE noredo; } exit EOF
Use the below sql to check the status of recovery

select trim(round(sum(sofar)/sum(totalwork)*100,2)) from v$session_longops where sid in (select sid from v$session where program like 'rman@%' and status='ACTIVE') and to_char(start_time,'DD-MON-YYYY')='01-FEB-2011' and sofar != TOTALWORK;
Once recovery gets completed, 1. 2. 3. 4. Recover the DR/DG till the standby controlfile creation time as per STEP 3. Stop DR/DG database and revert back the controlfile_record_keep_time parameter Open the DR/DG in READ ONLY MODE to verify the DB is good Revert back to the normal procedure to sync the DR/DG database.

REFERENCES 1. Oracle Database Backup and Recovery Advanced User's Guide 10g Release 2 (10.2) - B14191-03. Pg 13-24 - Using RMAN Incremental Backups to Refresh a Standby Database 2. Oracle SR Raised for C3 Standby database Issue : 3-2880116901 3. Oracle Note Id - 397269.1 , Relation between RMAN retention period and control_file_record_keep_time