ASM HANDS-ON TRAINING

Lab 13 ASM and Rman Crash, Restore and Recovery Scenarios

Alejandro Vargas | Principal Support Consultant
Oracle Advanced Customer Services

INDEX
Summary.......................................................................................................................................................................3
Setup Archivelog Mode.................................................................................................................................................3
Full Rman Backup.........................................................................................................................................................5
Enable change tracking file...........................................................................................................................................6
Make A Full Incremental Hot Backup............................................................................................................................6
Checking the Backup..................................................................................................................................................11
Crash Restore and Recovery Scenarios.....................................................................................................................13
System tablespace loss...............................................................................................................................................13
Restore And Recover System Tablespace.................................................................................................................14
User datafile loss.........................................................................................................................................................18
....................................................................................................................................................................................21
Restore and Recover users tablespace......................................................................................................................21
Online redologs loss....................................................................................................................................................25
....................................................................................................................................................................................27
Restore and Recover from Online Redolog loss.........................................................................................................27
Restore And Recover From Controlfile Loss..............................................................................................................38
Total Database Loss ..................................................................................................................................................42
Restore And Recover From Total Loss.......................................................................................................................44
Scripts..........................................................................................................................................................................52
backup-incr-lev0-plus-recover.rmn..............................................................................................................................52
1-system-tablespace-loss............................................................................................................................................54
1-restore-from-system-tablespace-loss.......................................................................................................................55
2-user-datafile-loss......................................................................................................................................................56
2-restore-from-user-datafile-loss.................................................................................................................................58
3-online-redolog-loss...................................................................................................................................................59
3-restore-and-recover-from-online-redo-loss..............................................................................................................61
4-restore-and-recover-from-controlfile-loss.................................................................................................................62

1/67
 ASM HANDS-ON TRAINING
Lab 13 ASM and Rman Crash, Restore and Recovery Scenarios
5-total-database-loss...................................................................................................................................................63
5-restore-and-recover-from-total-database-loss.........................................................................................................65

2/67
 ASM HANDS-ON TRAINING
Lab 13 ASM and Rman Crash, Restore and Recovery Scenarios
Alejandro Vargas | Principal Support Consultant
Oracle Advanced Customer Services

Summary

On this Lab we will review the following crash and recovery scenarios, and we will implement some of them:

• System tablespace loss

• Recover system tablespace
• User datafile loss
• Recover users tablespace
• Online redo loss
• Recover from redo loss
• Controlfile loss
• Recover from controlfile loss
• Database loss
• Recover from total loss

Setup Archivelog Mode

The tests wil be done using archivelog mode and hot backups, then we will restart the database on archivelog mode.

[oracle@asmxpt app]$ ora

-------------
ORACLE_BASE=/u01/app/oracle
ORACLE_HOME=/u01/app/oracle/11g_db
ORACLE_SID=sati11
-------------
oracle 6462 1 0 09:58 ? 00:00:06 ora_smon_sati11
oracle 8904 1 0 Feb12 ? 00:00:02 asm_smon_+ASM

3/67
 ASM HANDS-ON TRAINING
Lab 13 ASM and Rman Crash, Restore and Recovery Scenarios
-------------
oracle 8693 1 0 Feb12 ? 00:00:18 /u01/app/oracle/11g_db/bin/tnslsnr
LISTENER_ASMXPT -inherit
-------------
[oracle@asmxpt app]$ sql

SQL*Plus: Release 11.1.0.7.0 - Production on Fri Feb 13 14:17:45 2009

Copyright (c) 1982, 2008, Oracle. All rights reserved.

Connected to:
Oracle Database 11g Enterprise Edition Release 11.1.0.7.0 - Production
With the Partitioning, OLAP, Data Mining and Real Application Testing options

SQL> shutdown immediate;

Database closed.
Database dismounted.
ORACLE instance shut down.
SQL> startup mount;
ORACLE instance started.

Total System Global Area 351522816 bytes

Fixed Size 1313344 bytes
Variable Size 268436928 bytes
Database Buffers 75497472 bytes
Redo Buffers 6275072 bytes
Database mounted.
SQL> alter database archivelog;

Database altered.

4/67
 ASM HANDS-ON TRAINING
Lab 13 ASM and Rman Crash, Restore and Recovery Scenarios

SQL> alter database open;

Database altered.

SQL> archive log list

Database log mode Archive Mode
Automatic archival Enabled
Archive destination USE_DB_RECOVERY_FILE_DEST
Oldest online log sequence 1
Next log sequence to archive 1
Current log sequence 1

SQL> exit
Disconnected from Oracle Database 11g Enterprise Edition Release 11.1.0.7.0 - Production
With the Partitioning, OLAP, Data Mining and Real Application Testing options

Full Rman Backup

Having the database set on archivelog mode we will make a full backup to the flash recovery area

SQL> show parameters recov

NAME TYPE VALUE

------------------------------------ ----------- ------------------------------
db_recovery_file_dest string +FRADG
db_recovery_file_dest_size big integer 6048M
recovery_parallelism integer 0

5/67
 ASM HANDS-ON TRAINING
Lab 13 ASM and Rman Crash, Restore and Recovery Scenarios
Enable change tracking file
To be able to make incremental backups that will quickly find only the changed blocks we will enable change tracking file.

SQL> ALTER DATABASE ENABLE BLOCK CHANGE TRACKING USING FILE '+fradg';

Database altered.

SQL> select * from v$BLOCK_CHANGE_TRACKING ;

STATUS FILENAME BYTES

---------- -------------------------------------------------- ----------
ENABLED +FRADG/sati11/changetracking/ctf.259.678750225 11599872

Make A Full Incremental Hot Backup

This backlup script is intended to make a backup that will be automatically updated with the last changes each time the
script is invoked.
The first time the script is called there is no level 0 backup to update so a full level 0 backup is created.
The second time all changed blocks since the previous backup are backed up
Every other execution the last incremental is applied to the backup and a new incremental is created

RUN
{
ALLOCATE CHANNEL disk1 DEVICE TYPE DISK FORMAT '+FRADG/%U';
ALLOCATE CHANNEL disk2 DEVICE TYPE DISK FORMAT '+FRADG/%U';
RECOVER COPY OF DATABASE
WITH TAG 'INCREMENTAL_DAILY_UPDATED' ;
BACKUP
INCREMENTAL LEVEL 1
FOR RECOVER OF COPY WITH TAG 'INCREMENTAL_DAILY_UPDATED'

6/67
 ASM HANDS-ON TRAINING
Lab 13 ASM and Rman Crash, Restore and Recovery Scenarios
DATABASE FORMAT '+FRADG/%U'
PLUS ARCHIVELOG FORMAT '+FRADG/%U';
}

[oracle@asmxpt ~]$ rman target /

Recovery Manager: Release 11.1.0.7.0 - Production on Sat Feb 14 04:07:31 2009

Copyright (c) 1982, 2007, Oracle. All rights reserved.

connected to target database: SATI11 (DBID=124143874)

RMAN> @backup-incr-lev0-plus-recover.rmn

RMAN> RUN
2> {
3> ALLOCATE CHANNEL disk1 DEVICE TYPE DISK ;
4> ALLOCATE CHANNEL disk2 DEVICE TYPE DISK ;
5> ALLOCATE CHANNEL disk3 DEVICE TYPE DISK ;
6> ALLOCATE CHANNEL disk4 DEVICE TYPE DISK ;
7> RECOVER COPY OF DATABASE
8> WITH TAG 'INCREMENTAL_DAILY_UPDATED' ;
9> BACKUP INCREMENTAL LEVEL 1
10> FOR RECOVER OF COPY WITH TAG 'INCREMENTAL_DAILY_UPDATED'
11> DATABASE PLUS ARCHIVELOG ;
12> }
using target database control file instead of recovery catalog
allocated channel: disk1
channel disk1: SID=130 device type=DISK

7/67
 ASM HANDS-ON TRAINING
Lab 13 ASM and Rman Crash, Restore and Recovery Scenarios
allocated channel: disk2
channel disk2: SID=131 device type=DISK

allocated channel: disk3

channel disk3: SID=127 device type=DISK

allocated channel: disk4

channel disk4: SID=128 device type=DISK

Starting recover at 14-FEB-09

no copy of datafile 1 found to recover
no copy of datafile 2 found to recover
no copy of datafile 3 found to recover
no copy of datafile 4 found to recover
Finished recover at 14-FEB-09

Starting backup at 14-FEB-09

current log archived
channel disk1: starting archived log backup set
channel disk1: specifying archived log(s) in backup set
input archived log thread=1 sequence=2 RECID=2 STAMP=678773278
channel disk1: starting piece 1 at 14-FEB-09
channel disk2: starting archived log backup set
channel disk2: specifying archived log(s) in backup set
input archived log thread=1 sequence=1 RECID=1 STAMP=678751046
channel disk2: starting piece 1 at 14-FEB-09
channel disk1: finished piece 1 at 14-FEB-09
piece handle=+FRADG/sati11/backupset/2009_02_14/annnf0_tag20090214t040758_0.260.678773279
tag=TAG20090214T040758 comment=NONE
channel disk1: backup set complete, elapsed time: 00:00:16

8/67
 ASM HANDS-ON TRAINING
Lab 13 ASM and Rman Crash, Restore and Recovery Scenarios
channel disk2: finished piece 1 at 14-FEB-09
piece handle=+FRADG/sati11/backupset/2009_02_14/annnf0_tag20090214t040758_0.261.678773281
tag=TAG20090214T040758 comment=NONE
channel disk2: backup set complete, elapsed time: 00:00:16
Finished backup at 14-FEB-09

Starting backup at 14-FEB-09

no parent backup or copy of datafile 1 found
no parent backup or copy of datafile 2 found
no parent backup or copy of datafile 3 found
no parent backup or copy of datafile 4 found
channel disk1: starting datafile copy
input datafile file number=00001 name=+DATADGNR/sati11/datafile/system.259.678706167
channel disk2: starting datafile copy
input datafile file number=00002 name=+DATADGNR/sati11/datafile/sysaux.261.678706169
channel disk3: starting datafile copy
input datafile file number=00003 name=+DATADGNR/sati11/datafile/undotbs1.260.678706169
channel disk4: starting datafile copy
input datafile file number=00004 name=+DATADGNR/sati11/datafile/users.268.678706171
output file name=+FRADG/sati11/datafile/users.265.678773315 tag=INCREMENTAL_DAILY_UPDATED
RECID=1 STAMP=678773321
channel disk4: datafile copy complete, elapsed time: 00:00:26
output file name=+FRADG/sati11/datafile/undotbs1.264.678773305
tag=INCREMENTAL_DAILY_UPDATED RECID=2 STAMP=678773528
channel disk3: datafile copy complete, elapsed time: 00:03:53
output file name=+FRADG/sati11/datafile/system.262.678773299 tag=INCREMENTAL_DAILY_UPDATED
RECID=4 STAMP=678773597
channel disk1: datafile copy complete, elapsed time: 00:05:05
output file name=+FRADG/sati11/datafile/sysaux.263.678773303 tag=INCREMENTAL_DAILY_UPDATED
RECID=3 STAMP=678773597
channel disk2: datafile copy complete, elapsed time: 00:05:05

9/67
 ASM HANDS-ON TRAINING
Lab 13 ASM and Rman Crash, Restore and Recovery Scenarios
Finished backup at 14-FEB-09

Starting backup at 14-FEB-09

current log archived
channel disk1: starting archived log backup set
channel disk1: specifying archived log(s) in backup set
input archived log thread=1 sequence=3 RECID=3 STAMP=678773603
channel disk1: starting piece 1 at 14-FEB-09
channel disk1: finished piece 1 at 14-FEB-09
piece handle=+FRADG/sati11/backupset/2009_02_14/annnf0_tag20090214t041323_0.267.678773605
tag=TAG20090214T041323 comment=NONE
channel disk1: backup set complete, elapsed time: 00:00:01
Finished backup at 14-FEB-09

Starting Control File and SPFILE Autobackup at 14-FEB-09

piece handle=+FRADG/sati11/autobackup/2009_02_14/s_678773606.268.678773609 comment=NONE
Finished Control File and SPFILE Autobackup at 14-FEB-09
released channel: disk1
released channel: disk2
released channel: disk3
released channel: disk4

RMAN> **end-of-file**

10/67
 ASM HANDS-ON TRAINING
Lab 13 ASM and Rman Crash, Restore and Recovery Scenarios

Checking the Backup

RMAN> list backupset;

using target database control file instead of recovery catalog

List of Backup Sets

===================

BS Key Size Device Type Elapsed Time Completion Time

------- ---------- ----------- ------------ ---------------
6 362.00K DISK 00:00:00 14-FEB-09
BP Key: 6 Status: AVAILABLE Compressed: NO Tag: TAG20090214T041323
Piece Name: +FRADG/sati11/backupset/2009_02_14/annnf0_tag20090214t041323_0.267.678773605

List of Archived Logs in backup set 6

Thrd Seq Low SCN Low Time Next SCN Next Time
---- ------- ---------- --------- ---------- ---------
1 3 612605 14-FEB-09 613088 14-FEB-09

BS Key Type LV Size Device Type Elapsed Time Completion Time

------- ---- -- ---------- ----------- ------------ ---------------
7 Full 9.36M DISK 00:00:03 14-FEB-09
BP Key: 7 Status: AVAILABLE Compressed: NO Tag: TAG20090214T041326
Piece Name: +FRADG/sati11/autobackup/2009_02_14/s_678773606.268.678773609
SPFILE Included: Modification time: 14-FEB-09
SPFILE db_unique_name: SATI11
Control File Included: Ckp SCN: 613098 Ckp time: 14-FEB-09

RMAN> list archivelog all;

11/67
 ASM HANDS-ON TRAINING
Lab 13 ASM and Rman Crash, Restore and Recovery Scenarios

List of Archived Log Copies for database with db_unique_name SATI11

=====================================================================

Key Thrd Seq S Low Time

------- ---- ------- - ---------
3 1 3 A 14-FEB-09
Name: +FRADG/sati11/archivelog/2009_02_14/thread_1_seq_3.266.678773603

Checking the backup directories on ASM

[oracle@asmxpt sati11]$ asmcmd ls -l fradg/sati11/AUTOBACKUP/2009_02_14/

Type Redund Striped Time Sys Name
AUTOBACKUP UNPROT COARSE FEB 14 04:00:00 Y s_678773606.268.678773609

[oracle@asmxpt sati11]$ asmcmd ls -l fradg/sati11/BACKUPSET/2009_02_14/

Type Redund Striped Time Sys Name
BACKUPSET UNPROT COARSE FEB 14 04:00:00 Y annnf0_TAG20090214T041323_0.267.678773605

12/67
 ASM HANDS-ON TRAINING
Lab 13 ASM and Rman Crash, Restore and Recovery Scenarios

Crash Restore and Recovery Scenarios

System tablespace loss

On the first part of this scenario we will crash the database and remove the system tablespace, we will try to get the
database up…
The scenario is run using the script :

/home/oracle/CRASH-AND-RECOVERY-SCRIPTS/1-system-tablespace-loss

[oracle@asmxpt ~/CRASH-AND-RECOVERY-SCRIPTS]$ ./1-system-tablespace-loss

Generating database crash ...

System Tablespace Datafile is +DATADGNR/sati11/datafile/system.259.678779875

ORACLE instance shut down.

Generating system tablespace loss ...

State Type Rebal Sector Block AU Total_MB Free_MB Req_mir_free_MB Usable_file_MB Offline_disks Name
MOUNTED NORMAL N 512 4096 1048576 8188 3970 2047 961 0 DATADGNR/
MOUNTED EXTERN N 512 4096 1048576 6141 4401 0 4401 0 FRADG/

SYSAUX.261.678706169
SYSTEM.259.678779875
UNDOTBS1.260.678706169
USERS.268.678706171

State Type Rebal Sector Block AU Total_MB Free_MB Req_mir_free_MB Usable_file_MB Offline_disks Name

13/67
 ASM HANDS-ON TRAINING
Lab 13 ASM and Rman Crash, Restore and Recovery Scenarios
MOUNTED NORMAL N 512 4096 1048576 8188 5374 2047 1663 0 DATADGNR/
MOUNTED EXTERN N 512 4096 1048576 6141 4401 0 4401 0 FRADG/

SYSAUX.261.678706169
UNDOTBS1.260.678706169
USERS.268.678706171

Trying to restart the database after the crash ...

ORACLE instance started.

Total System Global Area 351522816 bytes

Fixed Size 1313344 bytes
Variable Size 272631232 bytes
Database Buffers 71303168 bytes
Redo Buffers 6275072 bytes
Database mounted.
ORA-01157: cannot identify/lock data file 1 - see DBWR trace file
ORA-01110: data file 1: '+DATADGNR/sati11/datafile/system.259.678779875'

Restore And Recover System Tablespace

The restore of a lost system tablespace is simple if we have a good backup; it is implemented running the following
commands:

• Restore datafile 1;
• Recover database;

The restore and recover from system tablespace loss is run using this script:

/home/oracle/CRASH-AND-RECOVERY-SCRIPTS/1- restore-from-system-tablespace-loss

14/67
 ASM HANDS-ON TRAINING
Lab 13 ASM and Rman Crash, Restore and Recovery Scenarios
[oracle@asmxpt ~/CRASH-AND-RECOVERY-SCRIPTS]$ ./1-restore-from-system-tablespace-loss

Executing Command : RESTORE and RECOVER SYSTEM DATAFILE

Recovery Manager: Release 11.1.0.7.0 - Production on Sat Feb 14 06:30:37 2009

Copyright (c) 1982, 2007, Oracle. All rights reserved.

connected to target database: SATI11 (DBID=124143874, not open)

RMAN>
Starting restore at 14-FEB-09
using target database control file instead of recovery catalog
allocated channel: ORA_DISK_1
channel ORA_DISK_1: SID=152 device type=DISK

channel ORA_DISK_1: restoring datafile 00001

input datafile copy RECID=4 STAMP=678773597 file
name=+FRADG/sati11/datafile/system.262.678773299
destination for restore of datafile 00001: +DATADGNR/sati11/datafile/system.259.678779875
channel ORA_DISK_1: copied datafile copy of datafile 00001
output file name=+DATADGNR/sati11/datafile/system.259.678781847 RECID=0 STAMP=0
Finished restore at 14-FEB-09

RMAN>
Starting recover at 14-FEB-09
using channel ORA_DISK_1

starting media recovery

15/67
 ASM HANDS-ON TRAINING
Lab 13 ASM and Rman Crash, Restore and Recovery Scenarios

archived log for thread 1 with sequence 3 is already on disk as file

+FRADG/sati11/archivelog/2009_02_14/thread_1_seq_3.266.678773603
archived log for thread 1 with sequence 4 is already on disk as file
+FRADG/sati11/archivelog/2009_02_14/thread_1_seq_4.261.678779175
archived log for thread 1 with sequence 5 is already on disk as file
+FRADG/sati11/archivelog/2009_02_14/thread_1_seq_5.260.678779567
archived log for thread 1 with sequence 6 is already on disk as file
+FRADG/sati11/archivelog/2009_02_14/thread_1_seq_6.257.678780233
archived log file name=+FRADG/sati11/archivelog/2009_02_14/thread_1_seq_3.266.678773603
thread=1 sequence=3
archived log file name=+FRADG/sati11/archivelog/2009_02_14/thread_1_seq_4.261.678779175
thread=1 sequence=4
media recovery complete, elapsed time: 00:00:06
Finished recover at 14-FEB-09

RMAN>
database opened

RMAN>

Recovery Manager complete.

SQL*Plus: Release 11.1.0.7.0 - Production on Sat Feb 14 06:33:11 2009

Copyright (c) 1982, 2008, Oracle. All rights reserved.

Connected to:
Oracle Database 11g Enterprise Edition Release 11.1.0.7.0 - Production
With the Partitioning, OLAP, Data Mining and Real Application Testing options

16/67
 ASM HANDS-ON TRAINING
Lab 13 ASM and Rman Crash, Restore and Recovery Scenarios

SQL>
FILE_NAME
--------------------------------------------------------------------------------
+DATADGNR/sati11/datafile/system.259.678781847
+DATADGNR/sati11/datafile/sysaux.261.678706169
+DATADGNR/sati11/datafile/undotbs1.260.678706169
+DATADGNR/sati11/datafile/users.268.678706171

SQL> Disconnected from Oracle Database 11g Enterprise Edition Release 11.1.0.7.0 -
Production
With the Partitioning, OLAP, Data Mining and Real Application Testing options

17/67
 ASM HANDS-ON TRAINING
Lab 13 ASM and Rman Crash, Restore and Recovery Scenarios

User datafile loss

On this scenario the database does not crash, we will set the users tablespace offline to be able to remove it from ASM.
Restore and recovery can also take place while the database is open.
When the application is executed the following error is returned:

ERROR at line 1:
ORA-00376: file 4 cannot be read at this time
ORA-01110: data file 4: '+DATADGNR/sati11/datafile/users.268.678706171'

The scenario is run using this script :

/home/oracle/CRASH-AND-RECOVERY-SCRIPTS/2-user-datafile-loss

[oracle@asmxpt ~/CRASH-AND-RECOVERY-SCRIPTS]$ ./2-user-datafile-loss

Preparing User Application ...

User dropped.
User created.
Grant succeeded.
Table created.
31 rows updated.
Commit complete.

USERNAME
------------------------------
CUSTOMER_SYSTEM
CUSTOMER_SYS
CUSTOMER_MGMT_VIEW

18/67
 ASM HANDS-ON TRAINING
Lab 13 ASM and Rman Crash, Restore and Recovery Scenarios
CUSTOMER_SYSMAN
CUSTOMER_DBSNMP
CUSTOMER_APPS
CUSTOMER_OUTLN
CUSTOMER_WKSYS
CUSTOMER_OLAPSYS
CUSTOMER_SI_INFORMTN_SCHEMA
CUSTOMER_OWBSYS
CUSTOMER_FLOWS_030000
CUSTOMER_ORDPLUGINS
CUSTOMER_WKPROXY
CUSTOMER_XDB
CUSTOMER_ANONYMOUS
CUSTOMER_CTXSYS
CUSTOMER_WK_TEST
CUSTOMER_WMSYS
CUSTOMER_EXFSYS
CUSTOMER_ORDSYS
CUSTOMER_MDSYS
CUSTOMER_FLOWS_FILES
CUSTOMER_SPATIAL_WFS_ADMIN_USR
CUSTOMER_SPATIAL_CSW_ADMIN_USR
CUSTOMER_APEX_PUBLIC_USER
CUSTOMER_DIP
CUSTOMER_MDDATA
CUSTOMER_XS$NULL
CUSTOMER_TSMSYS
CUSTOMER_ORACLE_OCM

31 rows selected.

19/67
 ASM HANDS-ON TRAINING
Lab 13 ASM and Rman Crash, Restore and Recovery Scenarios
Generating user datafile remove ...
Setting tablespace users offline ...

Tablespace altered.

Removing tablespace users datafile ...

State Type Rebal Sector Block AU Total_MB Free_MB Req_mir_free_MB Usable_file_MB Offline_disks Name
MOUNTED NORMAL N 512 4096 1048576 8188 3970 2047 961 0 DATADGNR/
MOUNTED EXTERN N 512 4096 1048576 6141 4374 0 4374 0 FRADG/

SYSAUX.261.678706169
SYSTEM.259.678781847
UNDOTBS1.260.678706169
USERS.268.678706171

SYSAUX.261.678706169
SYSTEM.259.678781847
UNDOTBS1.260.678706169
State Type Rebal Sector Block AU Total_MB Free_MB Req_mir_free_MB Usable_file_MB Offline_disks Name
MOUNTED NORMAL N 512 4096 1048576 8188 3982 2047 967 0 DATADGNR/
MOUNTED EXTERN N 512 4096 1048576 6141 4374 0 4374 0 FRADG/

Checking application ...

select username from apps.customers

*
ERROR at line 1:
ORA-00376: file 4 cannot be read at this time
ORA-01110: data file 4: '+DATADGNR/sati11/datafile/users.268.678706171'

20/67
 ASM HANDS-ON TRAINING
Lab 13 ASM and Rman Crash, Restore and Recovery Scenarios

Restore and Recover users tablespace

To restore and recover the users tablespace it is not necessary to bring the database down. The commands used in this
case from within Rman are:

• Restore tablespace users;

• Recover tablespace users;
• Alter tablespace users online

This script was used to execute the restore and recovery process:

/home/oracle/CRASH-AND-RECOVERY-SCRIPTS/2-restore-from-user-datafile-loss

[oracle@asmxpt ~/CRASH-AND-RECOVERY-SCRIPTS]$ ./2-restore-from-user-datafile-loss

Restoring and Recovering Tablespace Users ...

Recovery Manager: Release 11.1.0.7.0 - Production on Sat Feb 14 10:52:56 2009

Copyright (c) 1982, 2007, Oracle. All rights reserved.

connected to target database: SATI11 (DBID=124143874)

RMAN>
Starting restore at 14-FEB-09
using target database control file instead of recovery catalog
allocated channel: ORA_DISK_1
channel ORA_DISK_1: SID=128 device type=DISK

21/67
 ASM HANDS-ON TRAINING
Lab 13 ASM and Rman Crash, Restore and Recovery Scenarios

channel ORA_DISK_1: restoring datafile 00004

input datafile copy RECID=1 STAMP=678773321 file
name=+FRADG/sati11/datafile/users.265.678773315
destination for restore of datafile 00004: +DATADGNR/sati11/datafile/users.268.678706171
channel ORA_DISK_1: copied datafile copy of datafile 00004
output file name=+DATADGNR/sati11/datafile/users.268.678797585 RECID=0 STAMP=0
Finished restore at 14-FEB-09

RMAN>
Starting recover at 14-FEB-09
using channel ORA_DISK_1

starting media recovery

archived log for thread 1 with sequence 3 is already on disk as file

+FRADG/sati11/archivelog/2009_02_14/thread_1_seq_3.266.678773603
archived log for thread 1 with sequence 4 is already on disk as file
+FRADG/sati11/archivelog/2009_02_14/thread_1_seq_4.261.678779175
archived log for thread 1 with sequence 5 is already on disk as file
+FRADG/sati11/archivelog/2009_02_14/thread_1_seq_5.260.678779567
archived log for thread 1 with sequence 6 is already on disk as file
+FRADG/sati11/archivelog/2009_02_14/thread_1_seq_6.257.678780233
archived log for thread 1 with sequence 7 is already on disk as file
+FRADG/sati11/archivelog/2009_02_14/thread_1_seq_7.256.678781955
archived log file name=+FRADG/sati11/archivelog/2009_02_14/thread_1_seq_3.266.678773603
thread=1 sequence=3
archived log file name=+FRADG/sati11/archivelog/2009_02_14/thread_1_seq_4.261.678779175
thread=1 sequence=4
archived log file name=+FRADG/sati11/archivelog/2009_02_14/thread_1_seq_5.260.678779567
thread=1 sequence=5

22/67
 ASM HANDS-ON TRAINING
Lab 13 ASM and Rman Crash, Restore and Recovery Scenarios
media recovery complete, elapsed time: 00:00:05
Finished recover at 14-FEB-09

RMAN>
sql statement: alter tablespace users online

RMAN>

Recovery Manager complete.

Checking application ...

USERNAME
------------------------------
CUSTOMER_SYSTEM
CUSTOMER_SYS
CUSTOMER_MGMT_VIEW
CUSTOMER_SYSMAN
CUSTOMER_DBSNMP
CUSTOMER_APPS
CUSTOMER_OUTLN
CUSTOMER_WKSYS
CUSTOMER_OLAPSYS
CUSTOMER_SI_INFORMTN_SCHEMA
CUSTOMER_OWBSYS
CUSTOMER_FLOWS_030000
CUSTOMER_ORDPLUGINS
CUSTOMER_WKPROXY
CUSTOMER_XDB
CUSTOMER_ANONYMOUS
CUSTOMER_CTXSYS

23/67
 ASM HANDS-ON TRAINING
Lab 13 ASM and Rman Crash, Restore and Recovery Scenarios
CUSTOMER_WK_TEST
CUSTOMER_WMSYS
CUSTOMER_EXFSYS
CUSTOMER_ORDSYS
CUSTOMER_MDSYS
CUSTOMER_FLOWS_FILES
CUSTOMER_SPATIAL_WFS_ADMIN_USR
CUSTOMER_SPATIAL_CSW_ADMIN_USR
CUSTOMER_APEX_PUBLIC_USER
CUSTOMER_DIP
CUSTOMER_MDDATA
CUSTOMER_XS$NULL
CUSTOMER_TSMSYS
CUSTOMER_ORACLE_OCM

31 rows selected.

24/67
 ASM HANDS-ON TRAINING
Lab 13 ASM and Rman Crash, Restore and Recovery Scenarios

Online redologs loss

In this step we will simulate a database crash that led to loss the online redologs; when the database is restarted the
following errors are displayed:

ORA-00313: open failed for members of log group 1 of thread 1

ORA-00312: online log 1 thread 1:
'+DATADGNR/sati11/onlinelog/group_1.269.678706577'
ORA-00312: online log 1 thread 1:
'+DATADGNR/sati11/onlinelog/group_1.262.678706585'

This script was used to simulate the online redolog loss:

/home/oracle/CRASH-AND-RECOVERY-SCRIPTS/3-online-redolog-loss

[oracle@asmxpt ~/CRASH-AND-RECOVERY-SCRIPTS]$ ./3-online-redolog-loss

Generating database crash ...

asmcmd rm +DATADGNR/sati11/onlinelog/group_1.269.678706577
asmcmd rm +DATADGNR/sati11/onlinelog/group_1.262.678706585

Database log mode Archive Mode

Automatic archival Enabled
Archive destination USE_DB_RECOVERY_FILE_DEST
Oldest online log sequence 7
Next log sequence to archive 9
Current log sequence 9

SEQUENCE#

25/67
 ASM HANDS-ON TRAINING
Lab 13 ASM and Rman Crash, Restore and Recovery Scenarios
----------
8

GROUP# THREAD# SEQUENCE# BYTES MEMBERS ARC STATUS FIRST_CHANGE# FIRST_TIM

---------- ---------- ---------- ---------- ---------- --- ---------------- ------------- ---------
1 1 7 52428800 2 YES INACTIVE 676896 14-FEB-09
2 1 8 52428800 2 YES INACTIVE 708745 14-FEB-09
3 1 9 52428800 2 NO CURRENT 747092 14-FEB-09

Database closed.
Database dismounted.
ORACLE instance shut down.

Removing online redo log group ...

Starting database after the crash ...

ORACLE instance started.

Total System Global Area 351522816 bytes

Fixed Size 1313344 bytes
Variable Size 272631232 bytes
Database Buffers 71303168 bytes
Redo Buffers 6275072 bytes
Database mounted.
ORA-00313: open failed for members of log group 1 of thread 1
ORA-00312: online log 1 thread 1:
'+DATADGNR/sati11/onlinelog/group_1.269.678706577'
ORA-00312: online log 1 thread 1:
'+DATADGNR/sati11/onlinelog/group_1.262.678706585'

26/67
 ASM HANDS-ON TRAINING
Lab 13 ASM and Rman Crash, Restore and Recovery Scenarios

Restore and Recover from Online Redolog loss

In this step we perform the restore and recovery of the database when online redologs are missing.

When a redolog is missing a full database restore must be done.

The script will request the sequence # to recover using the until sequence clause. This sequence can be obtained from
the database alert.log

[oracle@asmxpt ~/CRASH-AND-RECOVERY-SCRIPTS]$ ./3-restore-and-recover-from-online-redo-loss

Executing Full Database Restore ...

Please check the last archived sequence of the database.

please enter sequence number to restore to ...
8

Recovery Manager: Release 11.1.0.7.0 - Production on Sat Feb 14 18:40:08 2009

Copyright (c) 1982, 2007, Oracle. All rights reserved.

connected to target database: SATI11 (DBID=124143874, not open)

RMAN>
database is already started

RMAN>
Starting restore at 14-FEB-09
using target database control file instead of recovery catalog

27/67
 ASM HANDS-ON TRAINING
Lab 13 ASM and Rman Crash, Restore and Recovery Scenarios
allocated channel: ORA_DISK_1
channel ORA_DISK_1: SID=145 device type=DISK

channel ORA_DISK_1: restoring datafile 00001

input datafile copy RECID=4 STAMP=678773597 file
name=+FRADG/sati11/datafile/system.262.678773299
destination for restore of datafile 00001: +DATADGNR/sati11/datafile/system.259.678781847
channel ORA_DISK_1: copied datafile copy of datafile 00001
output file name=+DATADGNR/sati11/datafile/system.259.678781847 RECID=0 STAMP=0
channel ORA_DISK_1: restoring datafile 00002
input datafile copy RECID=3 STAMP=678773597 file
name=+FRADG/sati11/datafile/sysaux.263.678773303
destination for restore of datafile 00002: +DATADGNR/sati11/datafile/sysaux.261.678706169
channel ORA_DISK_1: copied datafile copy of datafile 00002
output file name=+DATADGNR/sati11/datafile/sysaux.261.678706169 RECID=0 STAMP=0
channel ORA_DISK_1: restoring datafile 00003
input datafile copy RECID=2 STAMP=678773528 file
name=+FRADG/sati11/datafile/undotbs1.264.678773305
destination for restore of datafile 00003:
+DATADGNR/sati11/datafile/undotbs1.260.678706169
channel ORA_DISK_1: copied datafile copy of datafile 00003
output file name=+DATADGNR/sati11/datafile/undotbs1.260.678706169 RECID=0 STAMP=0
channel ORA_DISK_1: restoring datafile 00004
input datafile copy RECID=1 STAMP=678773321 file
name=+FRADG/sati11/datafile/users.265.678773315
destination for restore of datafile 00004: +DATADGNR/sati11/datafile/users.268.678797585
channel ORA_DISK_1: copied datafile copy of datafile 00004
output file name=+DATADGNR/sati11/datafile/users.268.678797585 RECID=0 STAMP=0
Finished restore at 14-FEB-09

RMAN>

28/67
 ASM HANDS-ON TRAINING
Lab 13 ASM and Rman Crash, Restore and Recovery Scenarios
Starting recover at 14-FEB-09
using channel ORA_DISK_1

starting media recovery

archived log for thread 1 with sequence 3 is already on disk as file

+FRADG/sati11/archivelog/2009_02_14/thread_1_seq_3.266.678773603
archived log for thread 1 with sequence 4 is already on disk as file
+FRADG/sati11/archivelog/2009_02_14/thread_1_seq_4.261.678779175
archived log for thread 1 with sequence 5 is already on disk as file
+FRADG/sati11/archivelog/2009_02_14/thread_1_seq_5.260.678779567
archived log for thread 1 with sequence 6 is already on disk as file
+FRADG/sati11/archivelog/2009_02_14/thread_1_seq_6.257.678780233
archived log for thread 1 with sequence 7 is already on disk as file
+FRADG/sati11/archivelog/2009_02_14/thread_1_seq_7.256.678781955
archived log file name=+FRADG/sati11/archivelog/2009_02_14/thread_1_seq_3.266.678773603
thread=1 sequence=3
archived log file name=+FRADG/sati11/archivelog/2009_02_14/thread_1_seq_4.261.678779175
thread=1 sequence=4
archived log file name=+FRADG/sati11/archivelog/2009_02_14/thread_1_seq_5.260.678779567
thread=1 sequence=5
archived log file name=+FRADG/sati11/archivelog/2009_02_14/thread_1_seq_6.257.678780233
thread=1 sequence=6
archived log file name=+FRADG/sati11/archivelog/2009_02_14/thread_1_seq_7.256.678781955
thread=1 sequence=7
media recovery complete, elapsed time: 00:00:19
Finished recover at 14-FEB-09

RMAN>
database opened

29/67
 ASM HANDS-ON TRAINING
Lab 13 ASM and Rman Crash, Restore and Recovery Scenarios
RMAN>

Recovery Manager complete.

Checking Database after online redolog loss and database restore and recover

SQL*Plus: Release 11.1.0.7.0 - Production on Sat Feb 14 18:46:31 2009

Copyright (c) 1982, 2008, Oracle. All rights reserved.

Connected to:
Oracle Database 11g Enterprise Edition Release 11.1.0.7.0 - Production
With the Partitioning, OLAP, Data Mining and Real Application Testing options

SQL> SQL> SQL>

GROUP# THREAD# SEQUENCE# BYTES MEMBERS ARC STATUS FIRST_CHANGE# FIRST_TIM
---------- ---------- ---------- ---------- ---------- --- ---------------- ------------- ---------
1 1 1 52428800 2 NO CURRENT 708746 14-FEB-09
2 1 0 52428800 2 YES UNUSED 0
3 1 0 52428800 2 YES UNUSED 0

SQL>
MEMBER
------------------------------------------------
+DATADGNR/sati11/onlinelog/group_1.262.678825909
+DATADGNR/sati11/onlinelog/group_1.269.678825917
+DATADGNR/sati11/onlinelog/group_2.264.678825925
+DATADGNR/sati11/onlinelog/group_2.263.678825931
+DATADGNR/sati11/onlinelog/group_3.266.678825939
+DATADGNR/sati11/onlinelog/group_3.265.678825943

6 rows selected.

30/67
 ASM HANDS-ON TRAINING
Lab 13 ASM and Rman Crash, Restore and Recovery Scenarios
SQL> Disconnected from Oracle Database 11g Enterprise Edition Release 11.1.0.7.0 -
Production
With the Partitioning, OLAP, Data Mining and Real Application Testing options

Before starting the new scenario we will make a full backup

[oracle@asmxpt ~/RMAN-SCRIPTS]$ rman target /

Recovery Manager: Release 11.1.0.7.0 - Production on Sat Feb 14 18:53:36 2009

Copyright (c) 1982, 2007, Oracle. All rights reserved.

connected to target database: SATI11 (DBID=124143874)

RMAN> @backup-incr-lev0-plus-recover.rmn

RMAN> # backup-incr-lev0-plus-recover.rmn
2> # ---------------------------------
3> # This script on the first run makes a level 0 backup if no one exists
4> # In the second run run makes an incremental level 1
5> # On every subsequent run:
6> # 1) Applies the previous level 1 to the backupset and
7> # 2) Creates a new level 1 backup
8> RUN
9> {
10> ALLOCATE CHANNEL disk1 DEVICE TYPE DISK ;
11> ALLOCATE CHANNEL disk2 DEVICE TYPE DISK ;
12> ALLOCATE CHANNEL disk3 DEVICE TYPE DISK ;
13> ALLOCATE CHANNEL disk4 DEVICE TYPE DISK ;
14> RECOVER COPY OF DATABASE
15> WITH TAG 'INCREMENTAL_DAILY_UPDATED' ;

31/67
 ASM HANDS-ON TRAINING
Lab 13 ASM and Rman Crash, Restore and Recovery Scenarios
16> BACKUP INCREMENTAL LEVEL 1
17> FOR RECOVER OF COPY WITH TAG 'INCREMENTAL_DAILY_UPDATED'
18> DATABASE PLUS ARCHIVELOG ;
19> }
using target database control file instead of recovery catalog
allocated channel: disk1
channel disk1: SID=126 device type=DISK

allocated channel: disk2

channel disk2: SID=132 device type=DISK

allocated channel: disk3

channel disk3: SID=127 device type=DISK

allocated channel: disk4

channel disk4: SID=133 device type=DISK

Starting recover at 14-FEB-09

Finished recover at 14-FEB-09

Starting backup at 14-FEB-09

current log archived
channel disk1: starting archived log backup set
channel disk1: specifying archived log(s) in backup set
input archived log thread=1 sequence=8 RECID=8 STAMP=678818997
channel disk1: starting piece 1 at 14-FEB-09
channel disk2: starting archived log backup set
channel disk2: specifying archived log(s) in backup set
input archived log thread=1 sequence=7 RECID=7 STAMP=678781959
channel disk2: starting piece 1 at 14-FEB-09

32/67
 ASM HANDS-ON TRAINING
Lab 13 ASM and Rman Crash, Restore and Recovery Scenarios
channel disk3: starting archived log backup set
channel disk3: specifying archived log(s) in backup set
input archived log thread=1 sequence=3 RECID=3 STAMP=678773603
input archived log thread=1 sequence=4 RECID=4 STAMP=678779177
channel disk3: starting piece 1 at 14-FEB-09
channel disk4: starting archived log backup set
channel disk4: specifying archived log(s) in backup set
input archived log thread=1 sequence=1 RECID=10 STAMP=678826433
channel disk4: starting piece 1 at 14-FEB-09
channel disk1: finished piece 1 at 14-FEB-09
piece handle=+FRADG/sati11/backupset/2009_02_14/annnf0_tag20090214t185354_0.274.678826435
tag=TAG20090214T185354 comment=NONE
channel disk1: backup set complete, elapsed time: 00:00:11
channel disk1: starting archived log backup set
channel disk1: specifying archived log(s) in backup set
input archived log thread=1 sequence=5 RECID=5 STAMP=678779566
input archived log thread=1 sequence=6 RECID=6 STAMP=678780233
channel disk1: starting piece 1 at 14-FEB-09
channel disk2: finished piece 1 at 14-FEB-09
piece handle=+FRADG/sati11/backupset/2009_02_14/annnf0_tag20090214t185354_0.275.678826437
tag=TAG20090214T185354 comment=NONE
channel disk2: backup set complete, elapsed time: 00:00:11
channel disk2: starting archived log backup set
channel disk2: specifying archived log(s) in backup set
input archived log thread=1 sequence=9 RECID=9 STAMP=678825907
channel disk2: starting piece 1 at 14-FEB-09
channel disk3: finished piece 1 at 14-FEB-09
piece handle=+FRADG/sati11/backupset/2009_02_14/annnf0_tag20090214t185354_0.276.678826441
tag=TAG20090214T185354 comment=NONE
channel disk3: backup set complete, elapsed time: 00:00:08
channel disk4: finished piece 1 at 14-FEB-09

33/67
 ASM HANDS-ON TRAINING
Lab 13 ASM and Rman Crash, Restore and Recovery Scenarios
piece handle=+FRADG/sati11/backupset/2009_02_14/annnf0_tag20090214t185354_0.277.678826443
tag=TAG20090214T185354 comment=NONE
channel disk4: backup set complete, elapsed time: 00:00:07
channel disk1: finished piece 1 at 14-FEB-09
piece handle=+FRADG/sati11/backupset/2009_02_14/annnf0_tag20090214t185354_0.278.678826447
tag=TAG20090214T185354 comment=NONE
channel disk1: backup set complete, elapsed time: 00:00:01
channel disk2: finished piece 1 at 14-FEB-09
piece handle=+FRADG/sati11/backupset/2009_02_14/annnf0_tag20090214t185354_0.279.678826449
tag=TAG20090214T185354 comment=NONE
channel disk2: backup set complete, elapsed time: 00:00:02
Finished backup at 14-FEB-09

Starting backup at 14-FEB-09

channel disk1: starting incremental level 1 datafile backup set
channel disk1: specifying datafile(s) in backup set
input datafile file number=00001 name=+DATADGNR/sati11/datafile/system.259.678781847
channel disk1: starting piece 1 at 14-FEB-09
channel disk2: starting incremental level 1 datafile backup set
channel disk2: specifying datafile(s) in backup set
input datafile file number=00002 name=+DATADGNR/sati11/datafile/sysaux.261.678706169
channel disk2: starting piece 1 at 14-FEB-09
channel disk3: starting incremental level 1 datafile backup set
channel disk3: specifying datafile(s) in backup set
input datafile file number=00003 name=+DATADGNR/sati11/datafile/undotbs1.260.678706169
channel disk3: starting piece 1 at 14-FEB-09
channel disk4: starting incremental level 1 datafile backup set
channel disk4: specifying datafile(s) in backup set
input datafile file number=00004 name=+DATADGNR/sati11/datafile/users.268.678797585
channel disk4: starting piece 1 at 14-FEB-09
channel disk4: finished piece 1 at 14-FEB-09

34/67
 ASM HANDS-ON TRAINING
Lab 13 ASM and Rman Crash, Restore and Recovery Scenarios
piece handle=+FRADG/sati11/backupset/2009_02_14/nnndn1_tag20090214t185410_0.283.678826477
tag=TAG20090214T185410 comment=NONE
channel disk4: backup set complete, elapsed time: 00:01:05
channel disk3: finished piece 1 at 14-FEB-09
piece handle=+FRADG/sati11/backupset/2009_02_14/nnndn1_tag20090214t185410_0.282.678826473
tag=TAG20090214T185410 comment=NONE
channel disk3: backup set complete, elapsed time: 00:02:10
channel disk1: finished piece 1 at 14-FEB-09
piece handle=+FRADG/sati11/backupset/2009_02_14/nnndn1_tag20090214t185410_0.280.678826453
tag=TAG20090214T185410 comment=NONE
channel disk1: backup set complete, elapsed time: 00:02:20
channel disk2: finished piece 1 at 14-FEB-09
piece handle=+FRADG/sati11/backupset/2009_02_14/nnndn1_tag20090214t185410_0.281.678826459
tag=TAG20090214T185410 comment=NONE
channel disk2: backup set complete, elapsed time: 00:02:20
Finished backup at 14-FEB-09

Starting backup at 14-FEB-09

current log archived
channel disk1: starting archived log backup set
channel disk1: specifying archived log(s) in backup set
input archived log thread=1 sequence=2 RECID=11 STAMP=678826592
channel disk1: starting piece 1 at 14-FEB-09
channel disk1: finished piece 1 at 14-FEB-09
piece handle=+FRADG/sati11/backupset/2009_02_14/annnf0_tag20090214t185632_0.285.678826593
tag=TAG20090214T185632 comment=NONE
channel disk1: backup set complete, elapsed time: 00:00:01
Finished backup at 14-FEB-09

Starting Control File and SPFILE Autobackup at 14-FEB-09

piece handle=+FRADG/sati11/autobackup/2009_02_14/s_678826594.286.678826597 comment=NONE

35/67
 ASM HANDS-ON TRAINING
Lab 13 ASM and Rman Crash, Restore and Recovery Scenarios
Finished Control File and SPFILE Autobackup at 14-FEB-09
released channel: disk1
released channel: disk2
released channel: disk3
released channel: disk4

RMAN> # -------------------------------------
2> # eof backup-incr-lev0-plus-recover.rmn
3> **end-of-file**

36/67
 ASM HANDS-ON TRAINING
Lab 13 ASM and Rman Crash, Restore and Recovery Scenarios
Controlfile loss

This scenario simulates a database crash that lead to a missing controlfile, when the database is restarted the following
error is displayed:

ORA-00205: error in identifying control file, check alert log for more info

The script that generates this scenario is this:

/home/oracle/CRASH-AND-RECOVERY-SCRIPTS/4-controlfile-loss

[oracle@asmxpt ~/CRASH-AND-RECOVERY-SCRIPTS]$ ./4-controlfile-loss

Generating database crash ...

asmcmd rm +DATADGNR/sati11/controlfile/current.258.678705843
asmcmd rm +DATADGNR/sati11/controlfile/current.257.678705845

Database closed.
Database dismounted.
ORACLE instance shut down.
Checking databases up ...
oracle 4146 1 0 11:35 ? 00:00:00 asm_smon_+ASM

Removing controlfiles ...

Starting databases after the crash ...

ORACLE instance started.

Total System Global Area 351522816 bytes

37/67
 ASM HANDS-ON TRAINING
Lab 13 ASM and Rman Crash, Restore and Recovery Scenarios
Fixed Size 1313344 bytes
Variable Size 272631232 bytes
Database Buffers 71303168 bytes
Redo Buffers 6275072 bytes
ORA-00205: error in identifying control file, check alert log for more info

Restore And Recover From Controlfile Loss

To restore the controlfile we will need to set the DBID on rman. You can check the DBID of the database on the backup
directory we used to build this database: /u01/app/oracle/BACKUP/sati11/dbid_124143874.lst.gz

[oracle@asmxpt ~/CRASH-AND-RECOVERY-SCRIPTS]$ ./4-restore-and-recover-from-controlfile-

loss

Executing Controlfile Restore

Please check the DBID on this file:

-rw-r--r-- 1 oracle dba 104 Feb 12 18:08

/u01/app/oracle/BACKUP/sati11/dbid_124143874.lst.gz

please enter DBID number of the database to restore the controlfile

124143874

Recovery Manager: Release 11.1.0.7.0 - Production on Sat Feb 14 20:27:53 2009

Copyright (c) 1982, 2007, Oracle. All rights reserved.

connected to target database: SATI11 (not mounted)

38/67
 ASM HANDS-ON TRAINING
Lab 13 ASM and Rman Crash, Restore and Recovery Scenarios
RMAN>
executing command: SET DBID

RMAN>
database is already started

RMAN>
Starting restore at 14-FEB-09
using target database control file instead of recovery catalog
allocated channel: ORA_DISK_1
channel ORA_DISK_1: SID=156 device type=DISK

recovery area destination: +FRADG

database name (or database unique name) used for search: SATI11
channel ORA_DISK_1: AUTOBACKUP
+fradg/SATI11/AUTOBACKUP/2009_02_14/s_678826594.286.678826597 found in the recovery area
channel ORA_DISK_1: looking for AUTOBACKUP on day: 20090214
channel ORA_DISK_1: restoring control file from AUTOBACKUP
+fradg/SATI11/AUTOBACKUP/2009_02_14/s_678826594.286.678826597
channel ORA_DISK_1: control file restore from AUTOBACKUP complete
output file name=+DATADGNR/sati11/controlfile/current.257.678832083
output file name=+DATADGNR/sati11/controlfile/current.258.678832085
Finished restore at 14-FEB-09

RMAN>
database mounted
released channel: ORA_DISK_1

RMAN>
Starting recover at 14-FEB-09
Starting implicit crosscheck backup at 14-FEB-09

39/67
 ASM HANDS-ON TRAINING
Lab 13 ASM and Rman Crash, Restore and Recovery Scenarios
allocated channel: ORA_DISK_1
channel ORA_DISK_1: SID=156 device type=DISK
Crosschecked 16 objects
Finished implicit crosscheck backup at 14-FEB-09

Starting implicit crosscheck copy at 14-FEB-09

using channel ORA_DISK_1
Crosschecked 4 objects
Finished implicit crosscheck copy at 14-FEB-09

searching for all files in the recovery area

cataloging files...
cataloging done

List of Cataloged Files

=======================
File Name: +fradg/SATI11/ARCHIVELOG/2009_02_14/thread_1_seq_3.287.678828379
File Name: +fradg/SATI11/AUTOBACKUP/2009_02_14/s_678826594.286.678826597

using channel ORA_DISK_1

starting media recovery

archived log for thread 1 with sequence 3 is already on disk as file

+FRADG/sati11/archivelog/2009_02_14/thread_1_seq_3.287.678828379
archived log for thread 1 with sequence 4 is already on disk as file
+DATADGNR/sati11/onlinelog/group_1.262.678825909
archived log file name=+FRADG/sati11/archivelog/2009_02_14/thread_1_seq_3.287.678828379
thread=1 sequence=3
archived log file name=+DATADGNR/sati11/onlinelog/group_1.262.678825909 thread=1
sequence=4

40/67
 ASM HANDS-ON TRAINING
Lab 13 ASM and Rman Crash, Restore and Recovery Scenarios
media recovery complete, elapsed time: 00:00:01
Finished recover at 14-FEB-09

RMAN>
database opened

RMAN>

Recovery Manager complete.

Checking Database after controlfile loss and restore

SQL> select name from v$controlfile;

NAME
--------------------------------------------------------------------------------
+DATADGNR/sati11/controlfile/current.257.678832083
+DATADGNR/sati11/controlfile/current.258.678832085

41/67
 ASM HANDS-ON TRAINING
Lab 13 ASM and Rman Crash, Restore and Recovery Scenarios

Total Database Loss

This scenario simulates a crash that lead to the complete loss of the database, the whole database directory is removed
the ASM diskgroup DATADG, all online logs, controlfiles, spfile and datafiles are lost.

Before the crash is simulated a complete set of information, that is required to recover the database, is displayed on the
screen. Keep this information at hand to be used during the restore steps. This information includes controlfile and spfile
backupset pieces, DBID and last archived sequences.

After the crash when trying to open the database the following errors are returned:

ORA-01078: failure in processing system parameters

ORA-01565: error in identifying file '+DATADGNR/sati11/spfilesati11.ora'
ORA-17503: ksfdopn:2 Failed to open file +DATADGNR/sati11/spfilesati11.ora
ORA-15056: additional error message
ORA-17503: ksfdopn:DGOpenFile05 Failed to open file +DATADGNR/sati11/spfilesati11.ora
ORA-17503: ksfdopn:2 Failed to open file +DATADGNR/sati11/spfilesati11.ora
ORA-15173: entry 'sati11' does not exist in directory '/'
ORA-06512: at line 4

This is the script that executes this scenario:

/home/oracle/CRASH-AND-RECOVERY-SCRIPTS/5-total-database-loss

[oracle@asmxpt ~/CRASH-AND-RECOVERY-SCRIPTS]$ ./5-total-database-loss

Generating database crash ...

Piece Name: +FRADG/sati11/autobackup/2009_02_15/s_678882001.261.678882005

Piece Name: +FRADG/sati11/autobackup/2009_02_15/s_678884598.271.678884601

42/67
 ASM HANDS-ON TRAINING
Lab 13 ASM and Rman Crash, Restore and Recovery Scenarios

Database log mode Archive Mode

Automatic archival Enabled
Archive destination USE_DB_RECOVERY_FILE_DEST
Oldest online log sequence 3
Next log sequence to archive 5
Current log sequence 5

NAME DBID
--------- ----------
SATI11 124143874

SEQUENCE#
----------
4

GROUP# THREAD# SEQUENCE# BYTES MEMBERS ARC STATUS FIRST_CHANGE# FIRST_TIM

---------- ---------- ---------- ---------- ---------- --- ---------------- ------------- ---------
1 1 4 52428800 2 YES INACTIVE 774566 15-FEB-09
2 1 5 52428800 2 NO CURRENT 774613 15-FEB-09
3 1 3 52428800 2 YES INACTIVE 772993 15-FEB-09

asmcmd rm -rf +DATADGNR/SATI11

Database closed.
Database dismounted.
ORACLE instance shut down.

Preparing to crash and burn database ...

43/67
 ASM HANDS-ON TRAINING
Lab 13 ASM and Rman Crash, Restore and Recovery Scenarios

Starting database after the crash ...

ORA-01078: failure in processing system parameters

ORA-01565: error in identifying file '+DATADGNR/sati11/spfilesati11.ora'
ORA-17503: ksfdopn:2 Failed to open file +DATADGNR/sati11/spfilesati11.ora
ORA-15056: additional error message
ORA-17503: ksfdopn:DGOpenFile05 Failed to open file +DATADGNR/sati11/spfilesati11.ora
ORA-17503: ksfdopn:2 Failed to open file +DATADGNR/sati11/spfilesati11.ora
ORA-15173: entry 'sati11' does not exist in directory '/'
ORA-06512: at line 4

Evaluating damage ...

Listing directories on ASM Data diskgroup

Listing directories on ASM Flash Recovery Area diskgroup

ARCHIVELOG/
AUTOBACKUP/
BACKUPSET/
CHANGETRACKING/
DATAFILE/

Restore And Recover From Total Loss

On this step we will restore the database after a total loss.

44/67
 ASM HANDS-ON TRAINING
Lab 13 ASM and Rman Crash, Restore and Recovery Scenarios
In first term we will recreate the database metadata using the backup we used before to build the database.

In this case we will restore only the metadata for diskgroup DATADGNR; this is necessary to be able to restore the spfile
and controlfile on the first stages of the restore procedure.

The instance is initially started by rman with dummy parameters; once the spfile is restored the instance is restarted in
nomount mode using the spfile.

Once the controlfile is successfully restored the database can be restored, recovered and opened with the resetlogs
option.

This is the script that implement the restore and recovery steps:

/home/oracle/CRASH-AND-RECOVERY-SCRIPTS/5-restore-and-recover-from-total-database-loss

[oracle@asmxpt ~/CRASH-AND-RECOVERY-SCRIPTS]$ ./5-restore-and-recover-from-total-database-loss

Executing Controlfile Restore

please enter DBID number of the database to restore the controlfile

124143874

Please check the last archived sequence of the database.

please enter sequence number to restore to ...
4

Please enter the full path to the controlfile and spfile backup to restore them

+FRADG/sati11/autobackup/2009_02_15/s_678884598.271.678884601

Rebuilding Database Directory on Data Diskgroup

45/67
 ASM HANDS-ON TRAINING
Lab 13 ASM and Rman Crash, Restore and Recovery Scenarios

Current Diskgroup being restored: DATADGNR

System template TEMPFILE modified!
System template DUMPSET modified!
System template XTRANSPORT modified!
System template DATAFILE modified!
System template ARCHIVELOG modified!
System template ASM_STALE modified!
System template CHANGETRACKING modified!
System template ONLINELOG modified!
System template FLASHBACK modified!
System template AUTOBACKUP modified!
System template DATAGUARDCONFIG modified!
System template PARAMETERFILE modified!
System template CONTROLFILE modified!
System template BACKUPSET modified!
Directory +DATADGNR/SATI11 re-created!
Directory +DATADGNR/SATI11/CONTROLFILE re-created!
Directory +DATADGNR/SATI11/ONLINELOG re-created!
Directory +DATADGNR/SATI11/DATAFILE re-created!
Directory +DATADGNR/SATI11/TEMPFILE re-created!
Directory +DATADGNR/SATI11/PARAMETERFILE re-created!
SATI11/

Executing Rman Restore and Recovery Steps

Recovery Manager: Release 11.1.0.7.0 - Production on Sun Feb 15 11:19:08 2009

Copyright (c) 1982, 2007, Oracle. All rights reserved.

46/67
 ASM HANDS-ON TRAINING
Lab 13 ASM and Rman Crash, Restore and Recovery Scenarios
connected to target database (not started)

RMAN>
executing command: SET DBID

RMAN>
startup failed: ORA-01078: failure in processing system parameters
ORA-01565: error in identifying file '+DATADGNR/sati11/spfilesati11.ora'
ORA-17503: ksfdopn:2 Failed to open file +DATADGNR/sati11/spfilesati11.ora
ORA-15056: additional error message
ORA-17503: ksfdopn:DGOpenFile05 Failed to open file +DATADGNR/sati11/spfilesati11.ora
ORA-17503: ksfdopn:2 Failed to open file +DATADGNR/sati11/spfilesati11.ora
ORA-15173: entry 'spfilesati11.ora' does not exist in directory 'sati11'
ORA-06512: at line 4

starting Oracle instance without parameter file for retrieval of spfile

Oracle instance started

Total System Global Area 159019008 bytes

Fixed Size 1312052 bytes

Variable Size 67111628 bytes
Database Buffers 88080384 bytes
Redo Buffers 2514944 bytes

RMAN>
Starting restore at 15-FEB-09
using target database control file instead of recovery catalog
allocated channel: ORA_DISK_1
channel ORA_DISK_1: SID=98 device type=DISK

47/67
 ASM HANDS-ON TRAINING
Lab 13 ASM and Rman Crash, Restore and Recovery Scenarios
channel ORA_DISK_1: restoring spfile from AUTOBACKUP
+FRADG/sati11/autobackup/2009_02_15/s_678884598.271.678884601
channel ORA_DISK_1: SPFILE restore from AUTOBACKUP complete
Finished restore at 15-FEB-09

RMAN>
Oracle instance started

Total System Global Area 351522816 bytes

Fixed Size 1313344 bytes

Variable Size 272631232 bytes
Database Buffers 71303168 bytes
Redo Buffers 6275072 bytes

RMAN>
Starting restore at 15-FEB-09
allocated channel: ORA_DISK_1
channel ORA_DISK_1: SID=152 device type=DISK

channel ORA_DISK_1: restoring control file

channel ORA_DISK_1: restore complete, elapsed time: 00:00:09
output file name=+DATADGNR/sati11/controlfile/current.257.678885579
output file name=+DATADGNR/sati11/controlfile/current.265.678885581
Finished restore at 15-FEB-09

RMAN>
database mounted
released channel: ORA_DISK_1

RMAN> 2> 3> 4> 5>

48/67
 ASM HANDS-ON TRAINING
Lab 13 ASM and Rman Crash, Restore and Recovery Scenarios
executing command: SET until clause

Starting restore at 15-FEB-09

Starting implicit crosscheck backup at 15-FEB-09
allocated channel: ORA_DISK_1
channel ORA_DISK_1: SID=152 device type=DISK
Crosschecked 10 objects
Finished implicit crosscheck backup at 15-FEB-09

Starting implicit crosscheck copy at 15-FEB-09

using channel ORA_DISK_1
Crosschecked 4 objects
Finished implicit crosscheck copy at 15-FEB-09

searching for all files in the recovery area

cataloging files...
cataloging done

List of Cataloged Files

=======================
File Name: +fradg/SATI11/AUTOBACKUP/2009_02_15/s_678884598.271.678884601

using channel ORA_DISK_1

channel ORA_DISK_1: restoring datafile 00001

input datafile copy RECID=19 STAMP=678881962 file
name=+FRADG/sati11/datafile/system.262.678773299
destination for restore of datafile 00001: +DATADGNR/sati11/datafile/system.260.678880089
channel ORA_DISK_1: copied datafile copy of datafile 00001
output file name=+DATADGNR/sati11/datafile/system.259.678885603 RECID=0 STAMP=0
channel ORA_DISK_1: restoring datafile 00002

49/67
 ASM HANDS-ON TRAINING
Lab 13 ASM and Rman Crash, Restore and Recovery Scenarios
input datafile copy RECID=20 STAMP=678881962 file
name=+FRADG/sati11/datafile/sysaux.263.678773303
destination for restore of datafile 00002: +DATADGNR/sati11/datafile/sysaux.261.678880175
channel ORA_DISK_1: copied datafile copy of datafile 00002
output file name=+DATADGNR/sati11/datafile/sysaux.261.678885687 RECID=0 STAMP=0
channel ORA_DISK_1: restoring datafile 00003
input datafile copy RECID=18 STAMP=678881962 file
name=+FRADG/sati11/datafile/undotbs1.264.678773305
destination for restore of datafile 00003:
+DATADGNR/sati11/datafile/undotbs1.259.678880249
channel ORA_DISK_1: copied datafile copy of datafile 00003
output file name=+DATADGNR/sati11/datafile/undotbs1.260.678885763 RECID=0 STAMP=0
channel ORA_DISK_1: restoring datafile 00004
input datafile copy RECID=17 STAMP=678881962 file
name=+FRADG/sati11/datafile/users.265.678773315
destination for restore of datafile 00004: +DATADGNR/sati11/datafile/users.265.678880295
channel ORA_DISK_1: copied datafile copy of datafile 00004
output file name=+DATADGNR/sati11/datafile/users.258.678885807 RECID=0 STAMP=0
Finished restore at 15-FEB-09

Starting recover at 15-FEB-09

using channel ORA_DISK_1

starting media recovery

archived log for thread 1 with sequence 2 is already on disk as file

+FRADG/sati11/archivelog/2009_02_15/thread_1_seq_2.279.678881853
archived log for thread 1 with sequence 3 is already on disk as file
+FRADG/sati11/archivelog/2009_02_15/thread_1_seq_3.276.678881967
archived log file name=+FRADG/sati11/archivelog/2009_02_15/thread_1_seq_2.279.678881853
thread=1 sequence=2

50/67
 ASM HANDS-ON TRAINING
Lab 13 ASM and Rman Crash, Restore and Recovery Scenarios
archived log file name=+FRADG/sati11/archivelog/2009_02_15/thread_1_seq_3.276.678881967
thread=1 sequence=3
media recovery complete, elapsed time: 00:00:08
Finished recover at 15-FEB-09

RMAN>
database opened

RMAN>

Recovery Manager complete.

Checking Database after total database loss, restore and recover

SQL*Plus: Release 11.1.0.7.0 - Production on Sun Feb 15 11:26:37 2009

Copyright (c) 1982, 2008, Oracle. All rights reserved.

Connected to:
Oracle Database 11g Enterprise Edition Release 11.1.0.7.0 - Production
With the Partitioning, OLAP, Data Mining and Real Application Testing options

SQL> SQL>
GROUP# THREAD# SEQUENCE# BYTES MEMBERS ARC STATUS FIRST_CHANGE# FIRST_TIM
---------- ---------- ---------- ---------- ---------- --- ---------------- ------------- ---------
1 1 1 52428800 2 NO CURRENT 774567 15-FEB-09
2 1 0 52428800 2 YES UNUSED 0
3 1 0 52428800 2 YES UNUSED 0

51/67
 ASM HANDS-ON TRAINING
Lab 13 ASM and Rman Crash, Restore and Recovery Scenarios
MEMBER
--------------------------------------------------------------------------------
+DATADGNR/sati11/onlinelog/group_1.262.678885823
+DATADGNR/sati11/onlinelog/group_1.269.678885829
+DATADGNR/sati11/onlinelog/group_2.264.678885837
+DATADGNR/sati11/onlinelog/group_2.263.678885845
+DATADGNR/sati11/onlinelog/group_3.266.678885917
+DATADGNR/sati11/onlinelog/group_3.268.678885927

NAME
---------
SATI11

SQL> Disconnected from Oracle Database 11g Enterprise Edition Release 11.1.0.7.0 -
Production
With the Partitioning, OLAP, Data Mining and Real Application Testing options

Scripts

backup-incr-lev0-plus-recover.rmn

# backup-incr-lev0-plus-recover.rmn
# ---------------------------------
# This script on the first run makes a level 0 backup if no one exists
# In the second run run makes an incremental level 1
# On every subsequent run:
# 1) Applies the previous level 1 to the backupset and

52/67
 ASM HANDS-ON TRAINING
Lab 13 ASM and Rman Crash, Restore and Recovery Scenarios
# 2) Creates a new level 1 backup
RUN
{
ALLOCATE CHANNEL disk1 DEVICE TYPE DISK ;
ALLOCATE CHANNEL disk2 DEVICE TYPE DISK ;
ALLOCATE CHANNEL disk3 DEVICE TYPE DISK ;
ALLOCATE CHANNEL disk4 DEVICE TYPE DISK ;
RECOVER COPY OF DATABASE
WITH TAG 'INCREMENTAL_DAILY_UPDATED' ;
BACKUP INCREMENTAL LEVEL 1
FOR RECOVER OF COPY WITH TAG 'INCREMENTAL_DAILY_UPDATED'
DATABASE PLUS ARCHIVELOG ;
}
# -------------------------------------
# eof backup-incr-lev0-plus-recover.rmn

53/67
 ASM HANDS-ON TRAINING
Lab 13 ASM and Rman Crash, Restore and Recovery Scenarios

1-system-tablespace-loss

#!/bin/tcsh
# 1-system-tablespace-loss
# ------------------------
# This script crash the database and removes the system tablespace
# ----------------------------------------------------------------
source ./set-environment
echo Generating database crash ...
echo
set v_rmf=`echo 'select file_name from dba_data_files where file_id=1;' | sqlplus -s / as
sysdba | grep system`
setenv ORACLE_SID +ASM
sqlplus -s $dbauser/$dbapwd as sysdba <<eof
shutdown abort;
eof
sqlplus -s $dbauser/$dbapwd as sysdba <<eof
startup
eof
echo
echo Generating system tablespace loss ...
echo
asmcmd lsdg
echo
asmcmd ls +datadgnr/sati11/datafile
echo
asmcmd rm -rf $v_rmf
echo
asmcmd lsdg
echo

54/67
 ASM HANDS-ON TRAINING
Lab 13 ASM and Rman Crash, Restore and Recovery Scenarios
asmcmd ls +datadgnr/sati11/datafile
echo
echo Trying to restart the database after the crash ...
echo
sqlplus -s $dbauser/$dbapwd@$datadb as sysdba <<eof
startup
eof
exit
# ----------------------------------------------------------------
# eof 1-system-tablespace-loss

1-restore-from-system-tablespace-loss

#!/bin/tcsh
# 1-restore-from-system-tablespace-loss
# ------------------------
# This script restore the system tablespace after a crash that removed the system
tablespace
# -------------------------------------------------------------------------------
source ./set-environment
echo
echo Executing Command : RESTORE and RECOVER SYSTEM DATAFILE
echo
setenv ORACLE_SID $datadb
rman TARGET / <<eof
restore datafile 1;
recover datafile 1;
alter database open;
eof
sqlplus $dbauser/$dbapwd@$datadb as sysdba <<eof

55/67
 ASM HANDS-ON TRAINING
Lab 13 ASM and Rman Crash, Restore and Recovery Scenarios
select file_name from dba_data_files;
exit
eof
exit
# ----------------------------------------------------------------
# eof 1-restore-from-system-tablespace-loss

2-user-datafile-loss

#!/bin/tcsh
# 2-user-datafile-loss
# ------------------------
# This script create an application user then removes the user tablespace datafile
# --------------------------------------------------------------------------------
source ./set-environment
setenv ORACLE_SID $datadb
clear
echo
echo Preparing User Application ...
echo
# connect to the database, create user a apps and a dummy table, select from it
# to simulate an application running
# -----------------------------------------------------------------------------
sqlplus -s $dbauser/$dbapwd@$datadb as sysdba <<eof
drop user apps cascade;
create user apps identified by apps default tablespace users temporary tablespace temp;
grant dba to apps;
connect apps/apps@$datadb
create table customers as select * from dba_users;
update customers set username='CUSTOMER_'||USERNAME;

56/67
 ASM HANDS-ON TRAINING
Lab 13 ASM and Rman Crash, Restore and Recovery Scenarios
commit;
select username from customers;
exit
eof
# generate the command to remove user's tablespace datafile from ASM
# the tablespace is set offline to be able to remove it
# -------------------------------------------------------------------
echo
echo Generating user datafile remove ...
echo
set v_rmf=`echo "select file_name from dba_data_files where tablespace_name='USERS';" |
sqlplus -s / as sysdba | grep users`
echo
echo Setting tablespace users offline ...
echo
sqlplus -s $dbauser/$dbapwd@$datadb as sysdba <<eof
alter tablespace users offline;
eof
# removing the users tablespace datafile from ASM
# -----------------------------------------------
echo
echo Removing tablespace users datafile ...
echo
setenv ORACLE_SID +ASM
asmcmd lsdg
echo
asmcmd ls +datadgnr/$datadb/datafile
echo
asmcmd rm -rf $v_rmf
echo
asmcmd ls +datadgnr/$datadb/datafile

57/67
 ASM HANDS-ON TRAINING
Lab 13 ASM and Rman Crash, Restore and Recovery Scenarios
echo
asmcmd lsdg
echo
echo Checking application ...
echo
sqlplus -s $dbauser/$dbapwd@$datadb as sysdba <<eof
select username from apps.customers;
exit
eof
# ----------------------------------------------------------------
# eof 2-user-datafile-loss

2-restore-from-user-datafile-loss

#!/bin/tcsh
# 2-restore-from-user-datafile-loss
# ------------------------
# This script restore the users tablespace while the database is still running
# ----------------------------------------------------------------------------
source ./set-environment
echo Restoring and Recovering Tablespace Users ...
echo
setenv ORACLE_SID $datadb
rman TARGET / <<eof
restore tablespace users;
recover tablespace users;
sql 'alter tablespace users online' ;
eof
echo
echo Checking application ...

58/67
 ASM HANDS-ON TRAINING
Lab 13 ASM and Rman Crash, Restore and Recovery Scenarios
echo
sqlplus -s $dbauser/$dbapwd@$datadb as sysdba <<eof
select username from apps.customers;
exit
eof
exit
# ----------------------------------------------------------------
# eof 2-restore-from-user-datafile-loss

3-online-redolog-loss

#!/bin/tcsh
# 3-online-redolog-loss
# ---------------------
# This script simulates a database crash followed by the loss of all
# online redologs
# ------------------------------------------------------------------
source ./set-environment
echo Generating database crash ...
echo
set v_logf=v\$logfile
set v_logs=v\$log
set v_logh=v\$log_history
setenv ORACLE_SID $datadb

# Create a script to remove online redologs

# -----------------------------------------
sqlplus -s / as sysdba <<eof
set pages 50000 lines 120 echo off head off veri off flush off ti off
spool rmonlnlog.sh
select 'asmcmd rm '||member||'' from $v_logf where group#=1;

59/67
 ASM HANDS-ON TRAINING
Lab 13 ASM and Rman Crash, Restore and Recovery Scenarios
spool off
set echo on head on veri on
set pages 50000 lines 120
spool redolog_miss_status-before_crash.log
archive log list;
select SEQUENCE# from $v_logh where FIRST_TIME=(select max(FIRST_TIME) from $v_logh);
select * from $v_logs;
spool off
SHUTDOWN IMMEDIATE;
eof
echo
# Remove online redologs
# ----------------------
echo Removing online redo log group ...
echo
setenv ORACLE_SID +ASM
chmod 700 ./rmonlnlog.sh
./rmonlnlog.sh
rm ./rmonlnlog.sh
# Restart the database after the simulated crash and check the error
# ------------------------------------------------------------------
echo
echo Starting database after the crash ...
echo
setenv ORACLE_SID $datadb
sqlplus -s / as sysdba <<eof
STARTUP
eof
# -------------------------
# eof 3-online-redolog-loss

60/67
 ASM HANDS-ON TRAINING
Lab 13 ASM and Rman Crash, Restore and Recovery Scenarios
3-restore-and-recover-from-online-redo-loss

#!/bin/tcsh
# 3-restore-and-recover-from-online-redo-loss
# -------------------------------------------
# this script executes the procedure to restore and recover the database
# after the loss of all online logs
# ----------------------------------------------------------------------
source ./set-environment
set v_logf=v\$logfile
set v_log=v\$log
echo Executing Full Database Restore ...
echo
echo Please check the last archived sequence of the database.
echo please enter sequence number to restore to ...
set v_seq = $<
echo
setenv ORACLE_SID $datadb
rman TARGET / <<eof
STARTUP MOUNT;
RESTORE DATABASE;
RECOVER DATABASE UNTIL SEQUENCE $v_seq THREAD 1;
ALTER DATABASE OPEN RESETLOGS;
eof
echo Checking Database after online redolog loss and database restore and recover
echo
sqlplus / as sysdba <<eof
set pages 50000 lines 200
col member for a55
select * from $v_log;
select member from $v_logf;

61/67
 ASM HANDS-ON TRAINING
Lab 13 ASM and Rman Crash, Restore and Recovery Scenarios
exit
eof
# -----------------------------------------------
# eof 3-restore-and-recover-from-online-redo-loss

4-restore-and-recover-from-controlfile-loss

#!/bin/tcsh
# 4-restore-and-recover-from-controlfile-loss
# -------------------------------------------
source ./set-environment
echo Executing Controlfile Restore
echo
echo Please check the DBID on this file:
echo
ls -l /u01/app/oracle/BACKUP/sati11/dbid_124143874.lst.gz
echo
echo please enter DBID number of the database to restore the controlfile
set v_dbid = $<
echo
# ------------------------------------------------
# Restore the controlfile and recover the database
# ------------------------------------------------
setenv ORACLE_SID $datadb
rman TARGET / <<eof
SET DBID $v_dbid;
STARTUP NOMOUNT;
RESTORE CONTROLFILE FROM AUTOBACKUP;
ALTER DATABASE MOUNT;
RECOVER DATABASE;
ALTER DATABASE OPEN RESETLOGS;

62/67
 ASM HANDS-ON TRAINING
Lab 13 ASM and Rman Crash, Restore and Recovery Scenarios
eof
set v_ctlf=v\$controlfile
echo Checking Database after controlfile loss and restore
echo
sqlplus / as sysdba <<eof
set pages 50000 lines 120
select name from $v_ctlf ;
exit
eof
# -----------------------------------------------
# eof 4-restore-and-recover-from-controlfile-loss

5-total-database-loss

#!/bin/tcsh
# 5-total-database-loss
# ---------------------
source ./set-environment
echo Generating database crash ...
echo
set v_par=v\$parameter
set v_dba=v\$database
set v_logs=v\$log
set v_logh=v\$log_history
setenv ORACLE_SID $datadb
# ------------------------------------------------
# Get backup to restore controlfile and spfile
# ------------------------------------------------
echo
rman target / @listbk | grep autobackup
echo

63/67
 ASM HANDS-ON TRAINING
Lab 13 ASM and Rman Crash, Restore and Recovery Scenarios
# ------------------------------------------------
# Get actual sequence before crashing the database
# ------------------------------------------------
sqlplus -s / as sysdba <<eof
set echo on head on veri on pages 50000 lines 120
spool redolog_miss_status-before_crash.log
archive log list;
select name,dbid from $v_dba;
select SEQUENCE# from $v_logh where FIRST_TIME=(select max(FIRST_TIME) from $v_logh);
select * from $v_logs;
spool off
set pages 50000 lines 120 echo off head off veri off flush off ti off
spool rmdbs.sh
select 'asmcmd rm -rf '||a.value||'/'||b.name
from $v_par a, $v_dba b
where a.name='db_create_file_dest';
spool off
SHUTDOWN IMMEDIATE
eof
# ------------------------------------------------
# Remove all database files
# ------------------------------------------------
echo
echo Preparing to crash and burn database ...
echo
setenv ORACLE_SID +ASM
chmod 700 ./rmdbs.sh
./rmdbs.sh
rm rmdbs.sh
echo
echo Starting database after the crash ...

64/67
 ASM HANDS-ON TRAINING
Lab 13 ASM and Rman Crash, Restore and Recovery Scenarios
echo
setenv ORACLE_SID $datadb
sqlplus -s / as sysdba <<eof
STARTUP
eof
# ------------------------------------------------
# Evaluate damage
# ------------------------------------------------
echo
echo Evaluating damage ...
echo
echo Listing directories on ASM Data diskgroup
echo
setenv ORACLE_SID +ASM
asmcmd ls +datadgnr
echo
echo Listing directories on ASM Flash Recovery Area diskgroup
echo
asmcmd ls +fradg/$datadb
echo
# -------------------------
# eof 5-total-database-loss

5-restore-and-recover-from-total-database-loss

#!/bin/tcsh
# --------------------------------------------------
# 5-restore-and-recover-from-total-database-loss
# --------------------------------------------------
source ./set-environment

65/67
 ASM HANDS-ON TRAINING
Lab 13 ASM and Rman Crash, Restore and Recovery Scenarios
echo Executing Controlfile Restore
echo
echo please enter DBID number of the database to restore the controlfile
set v_dbid = $<
echo
echo Please check the last archived sequence of the database.
echo please enter sequence number to restore to ...
set v_seq = $<
echo
echo Please enter the full path to the controlfile and spfile backup to restore them
echo
set v_bkp = $<
echo
# -----------------------------------
# Rebuild database diskgroup metadata
# -----------------------------------
echo Rebuilding Database Directory on Data Diskgroup
echo
setenv ORACLE_SID +ASM
asmcmd md_restore -b /u01/app/oracle/BACKUP/sati11/asm_md_backup -t nodg -g DATADGNR
asmcmd ls +DATADGNR
echo
echo Executing Rman Restore and Recovery Steps
echo
# -----------------------------------
# Restore the database
# -----------------------------------
setenv ORACLE_SID $datadb
rman TARGET / <<eof
SET DBID $v_dbid;
STARTUP NOMOUNT;

66/67
 ASM HANDS-ON TRAINING
Lab 13 ASM and Rman Crash, Restore and Recovery Scenarios
RESTORE SPFILE FROM '$v_bkp';
STARTUP FORCE NOMOUNT;
RESTORE CONTROLFILE FROM '$v_bkp';
ALTER DATABASE MOUNT;
run {
set until sequence $v_seq thread 1;
restore database;
recover database;
}
ALTER DATABASE OPEN RESETLOGS;
eof
set v_log=v\$log
set v_logf=v\$logfile
set v_dbs=v\$database
echo Checking Database after total database loss, restore and recover
echo
# --------------------------------------------
# Check the database after restore and recover
# --------------------------------------------
sqlplus $dbauser/$dbapwd@$datadb as sysdba <<eof
set pages 50000 lines 120
select * from $v_log;
elect member from $v_logf;
select name from $v_dbs;
exit
eof
# --------------------------------------------------
# eof 5-restore-and-recover-from-total-database-loss

End of Lab 13

67/67