Professional Documents
Culture Documents
GMT+05:30
SR 3-4526370651: Database hangs
completely
Information
Request
Number
3-4526370651
Primary
Contact
Vinay Kashyap
Status
Customer Working
Support ID
15917149
Opened
Product
Platform
Filed By
Primary
Contact
Phone
Severity
Legacy SR
Number
VINAY.KASHYAP@KOTAK.COM
+919820882572
1
History
Oracle Support - September 17, 2011 10:52:55 AM GMT+05:30 [Notes]
Generic Note
-----------------------Update
=======
Spoke to Customer's Manager Laxman 9873919042 and discussed the issue.
As per Laxman,the distributed transactions recovery was in progress during the issue time
Informed the following things,helps in issue investigations
+ Action plan given to previous update,for what to collect from database end when the hang
situation repeats
+ Advised to collect the OS information's like memory,io,cpu usage during the problem time
to review for OS scarcity if any like memory ,IO,swapspace etc
+ Advised to have a tab of nature of the activities in database prior to hang occurs
Thank You
Karunakar
Oracle Support - September 17, 2011 10:13:08 AM GMT+05:30 [Notes]
Generic Note
-----------------------Update
========
Spoke to customer at 0120 4726187 and discussed the issue.
+ So far the database appears hang for a duration of 10 - 15 mintues time
+ And during this time new connections or existing connections are not working
+ The hang situation was auto-corrected without any user intervension,
And new connections or existing connections are started working
+ Alert log contains jobs errors and ora-60 errors.
+ As per customer before the hang the cpu utilization was around 80 to 90 % ,during the hang
time the cpu utilization was dropped to around 20%
+ Advised customer to collect the following details when the hang situation repeats to
understand more about the cause.
Collect the 2 sets of hanganalyze + 2 sets of system state dumps during issue time
A)If you are able to connect to the DB during hang time,then use this steps
Open new sqlplus session each time:
SQL> connect / as sysdba
SQL> oradebug setmypid
SQL> oradebug unlimit
SQL> oradebug hanganalyze 3
SQL>oradebug tracefile_name
SQL>exit
... Wait for 1 minutes
Open new sqlplus session each time:
$ sqlplus /nolog
connect / as sysdba
oradebug setmypid
oradebug unlimit
SQL > ALTER SESSION SET EVENTS 'IMMEDIATE TRACE NAME SYSTEMSTATE
LEVEL 266';
wait 60 seconds
exit
FILE will output in the user_dump_dest directory....
Oracle Support - September 16, 2011 7:02:02 PM GMT+05:30 [ODM Data Collection]
Name
-------=== ODM Data Collection ===
=== ODM Data Collection ===
FileName
---------------alert log
FileComment
---------------------Numerous deadlock errors:
Example:
Fri Sep 16 15:46:07 2011
ORA-00060: Deadlock detected. More info in file
/oracle/product/admin/admin/klife/bdump/klife_j003_9138372.trc.
...
Fri Sep 16 16:48:51 2011
Shutting down instance (abort)
Fri Sep 16 16:49:01 2011
Instance termination failed to kill one or more processes
Instance terminated by USER, pid = 6209546
Fri Sep 16 16:50:39 2011
Starting ORACLE instance (normal)
sskgpgetexecname failed to get name
...
Fri Sep 16 16:50:48 2011
Completed redo scan
206145 redo blocks read, 8401 data blocks need recovery
Fri Sep 16 16:50:49 2011
Started redo application at
Fri Sep 16 16:51:02 2011
db_recovery_file_dest_size of 2048 MB is 0.00% used. This is a
user-specified limit on the amount of space that will be used by this
database for recovery-related files, and does not reflect the amount of
space available in the underlying filesystem or ASM diskgroup.
Fri Sep 16 16:51:02 2011
Completed: ALTER DATABASE OPEN