Tekelec 5200 Paramount Parkway Morrisville, NC 27560 888-FOR-TKLC

THIS DOCUMENT AND THE DATA DISCLOSED HEREIN OR HEREWITH IS PROPRIETARY AND IS NOT TO BE REPRODUCED, USED OR DISCLOSED IN WHOLE OR IN PART TO ANYONE WITHOUT THE WRITTEN PERMISSION OF TEKELEC. COPYRIGHT, TEKELEC 2007.

Title:

T1000 EPAP A - MO007185 - T1000 EPAP A - Dual Disk Replacement from 120GB to 250GB or 500GB Doc Number: MO007185 Revision: 1.0

TEKELEC
Method of Procedure

T1000 EPAP A - MO007185 - T1000 EPAP A - Dual Disk Replacement from 120GB to 250GB or 500GB

Title: Doc No.:

T1000 EPAP A - MO007185 - T1000 EPAP A - Dual Disk Replacement from 120GB to 250GB or 500GB MO007185 Revision: 1.0 Page 1 of 23

Tekelec 5200 Paramount Parkway Morrisville, NC 27560 888-FOR-TKLC

THIS DOCUMENT AND THE DATA DISCLOSED HEREIN OR HEREWITH IS PROPRIETARY AND IS NOT TO BE REPRODUCED, USED OR DISCLOSED IN WHOLE OR IN PART TO ANYONE WITHOUT THE WRITTEN PERMISSION OF TEKELEC. COPYRIGHT, TEKELEC 2007.

TEKELEC METHOD OF PROCEDURE

T1000 EPAP A - MO007185 - T1000 EPAP A - Dual Disk Replacement from 120GB to 250GB or 500GB , MO007185 General X Detailed Address: Office(s): Date: Start Date: Time: Completion Date: Time: General Description of Work: For dual disk replacement, run HDA and HDC replacement in any order and then execute the PARTITION EXPANSION procedure.

***Make note of which hard disk is failed and replace the failed hard disk first!!!!!!!
Additional Comments: Please review this MOP thoroughly, then sign this sheet and fax to 919-388-1384, ATTN: Ken Abbott, GlobalTAC Manager. This MOP must be signed before starting the maintenance activity for <Customer Name>.

***If this is the Active provisioning site, provisioning must be stopped and remain stopped until the MOP has been completed!!!
Attachments: None In Attendance Title Company Telephone #

Changes Required by Customer: ( ) Yes ( ) No This job has been reviewed and agreement reached. Responsibility for this job has been assigned to (Please Print): MOP Prepared by: Nitin Garg MOP Revisions by: Tony Fish MOP Tested by: MOP Test Comments: N/A Date: June 16, 2011 Date: November 22, 2011 Date:

No changes shall be made without the approval of the customer representative and the Tekelec representative. ***ESCALATION CALLOUT LIST*** For any anomaly that results from any step in this, Method Of Procedure, halt the process immediately. Escalate to the 1st authority from both Tekelec and the customer as listed below. Both parties must agree to problem resolution, and whether to proceed with the procedure TEKELEC CONTACT / NUMBER CUSTOMER CONTACT / NUMBER Preliminary TEKELEC Review by: Concurred in by TEKELEC: Approved by Customer Representative: Title: Doc No.: Title: Title: Global TAC Manager Title: Date: Date: Date:

T1000 EPAP A - MO007185 - T1000 EPAP A - Dual Disk Replacement from 120GB to 250GB or 500GB MO007185 Revision: 1.0 Page 2 of 23

Tekelec 5200 Paramount Parkway Morrisville, NC 27560 888-FOR-TKLC

THIS DOCUMENT AND THE DATA DISCLOSED HEREIN OR HEREWITH IS PROPRIETARY AND IS NOT TO BE REPRODUCED, USED OR DISCLOSED IN WHOLE OR IN PART TO ANYONE WITHOUT THE WRITTEN PERMISSION OF TEKELEC. COPYRIGHT, TEKELEC 2007.

Approved by Customer Representative:

Title:

Date:

General For dual disk replacement, run HDA and HDC replacement in any order and then execute PARTITION EXPANSION procedure. If one of the hard disk is faulty, the faulty hard disk MUST be replaced first!

PREPARE THE SYSTEM

S T E P # 1 This procedure outlines the steps for replacing both hard drives with new 500 GB drives on EPAP A running on T1000 hardware. Prior to work starting, Customer should: - If this is the Active provisioning site, the customer must stop the Provisioning system at this time. - If this is a provisioning site, perform PDBA Backups - Perform RTDB Backups - Turn off Automatic Backups login: root Password: <Enter root password> Should see: Last login: <date> from <hostname> Run uiEdit command on ACTIVE node. # uiEdit PDBA_REMOTE_PDBI_ALLOWED NO

Steps to be completed. Prepare system

2 3 4

Log into the MPS A as the root user. Verify output Block Provisioning (Step should be executed only in case of prov Server) Perform system check on MPS A. The syscheck utility should not report any problem other than a failed hard disk. Verify mirror status is good Verify all meta devices are in sync Stop Pdba software

# syscheck ***Make note of which hard disk is failed and replace the failed hard disk first!!!!!!! # cat /proc/mdstat ***Make note of which hard disk is failed and replace the failed hard disk first!!!!!!! Should not see recovery or resync. Each md should show active raid1 and [2/2] [UU]. # cd /etc/init.d Title: Doc No.: T1000 EPAP A - MO007185 - T1000 EPAP A - Dual Disk Replacement from 120GB to 250GB or 500GB MO007185 Revision: 1.0 Page 3 of 23

7 8

Tekelec 5200 Paramount Parkway Morrisville, NC 27560 888-FOR-TKLC

THIS DOCUMENT AND THE DATA DISCLOSED HEREIN OR HEREWITH IS PROPRIETARY AND IS NOT TO BE REPRODUCED, USED OR DISCLOSED IN WHOLE OR IN PART TO ANYONE WITHOUT THE WRITTEN PERMISSION OF TEKELEC. COPYRIGHT, TEKELEC 2007.

9 10 11 12

(Step should be executed only in case of prov Server) Verify output Stop Epap software Verify output Move Pdba and Epap executables to Pdba_stopped and Epap_stopped

# ./Pdba stop

Should see: PDBA application stopped. # ./Epap stop Should see: EPAP application stopped. # mv Epap Epap_stopped # mv Pdba Pdba_stopped

Title: Doc No.:

T1000 EPAP A - MO007185 - T1000 EPAP A - Dual Disk Replacement from 120GB to 250GB or 500GB MO007185 Revision: 1.0 Page 4 of 23

Tekelec 5200 Paramount Parkway Morrisville, NC 27560 888-FOR-TKLC

THIS DOCUMENT AND THE DATA DISCLOSED HEREIN OR HEREWITH IS PROPRIETARY AND IS NOT TO BE REPRODUCED, USED OR DISCLOSED IN WHOLE OR IN PART TO ANYONE WITHOUT THE WRITTEN PERMISSION OF TEKELEC. COPYRIGHT, TEKELEC 2007.

HDC REPLACEMENT
1 2 Prepare system Enter BIOS Please execute PREPARE THE SYSTEM procedure Upon restarting the server, enter the BIOS setup utility by pressing the DEL key when the message Press DEL to run Setup (F4 on Remote keyboard) is displayed. If using a PS2 keyboard (preferred method) the DEL key must be pressed immediately after the 3 keyboard LEDs light during the boot process. NOTE: This message is only displayed for a few seconds and the DEL or (F4 on Remote keyboard) key must be pressed before the message disappears from the screen or the boot sequence will continue as usual. The BIOS setup utility provides navigational tips which are displayed in the lower right hand corner of the screen, but in general moving around within the utility is accomplished using the right, left, up, and down arrow keys. To select the highlighted option, simply press the Enter key, and to return to the previous screen press the ESC key. If at anytime your output is garbled, in SecureCRT select Reset from the Edit menu. 3 Select Hard Disk Drives in the Boot Settings screen. Once inside the BIOS setup utility verify that the disk drive that is not being replaced is set to be the boot disk. The setting of interest is on the Boot settings screen within the BIOS setup utility. On the Boot settings screen select Hard Disk Drives as shown below. BIOS Boot Settings Screen

Title: Doc No.:

T1000 EPAP A - MO007185 - T1000 EPAP A - Dual Disk Replacement from 120GB to 250GB or 500GB MO007185 Revision: 1.0 Page 5 of 23

Tekelec 5200 Paramount Parkway Morrisville, NC 27560 888-FOR-TKLC

THIS DOCUMENT AND THE DATA DISCLOSED HEREIN OR HEREWITH IS PROPRIETARY AND IS NOT TO BE REPRODUCED, USED OR DISCLOSED IN WHOLE OR IN PART TO ANYONE WITHOUT THE WRITTEN PERMISSION OF TEKELEC. COPYRIGHT, TEKELEC 2007.

Verify that hda is listed as the 1st drive.

After selecting Hard Disk Drives, modify the drive order (the order of the entries can be reversed by pressing either the + or - key) to make the SM drive (hdc) the 1st Drive. NOTE: The device names listed on the Hard Disk Drives screen are the model number of the servers disk drives with one entry prepended by PM- (Primary Master) HDA and the other by SM- (Secondary Master) HDC. BIOS Hard Disk Drives Screen

Title: Doc No.:

T1000 EPAP A - MO007185 - T1000 EPAP A - Dual Disk Replacement from 120GB to 250GB or 500GB MO007185 Revision: 1.0 Page 6 of 23

Tekelec 5200 Paramount Parkway Morrisville, NC 27560 888-FOR-TKLC

THIS DOCUMENT AND THE DATA DISCLOSED HEREIN OR HEREWITH IS PROPRIETARY AND IS NOT TO BE REPRODUCED, USED OR DISCLOSED IN WHOLE OR IN PART TO ANYONE WITHOUT THE WRITTEN PERMISSION OF TEKELEC. COPYRIGHT, TEKELEC 2007.

Save any changes

If this setting requires modification make the necessary change, (the order of the entries can be reversed by pressing either the + or - key), and then save the CMOS data by pressing F10, and then selecting OK when the confirmation dialog is presented. If the order shown is correct, simply press the ESC key or select Discard Changes and Exit on the main Exit screen and select OK when asked to confirm the exit.

After exiting the BIOS setup utility, the server will restart. Once the login prompt is presented, log in as the root user. Verify output Perform system checks on MPS-A Verify no syscheck failures other than due to Pdba and Epap software being down. Verify mirror status is good

Login: root Password: <Enter root password>

7 8 9

Should see: Last login: <date> from <hostname> # syscheck The only failures should be in class proc module run due to the Pdba and Epap software being stopped. There will also be errors in connecting to the maint process and using the alarm dispatch utility. # cat /proc/mdstat Personalities : [raid1] read_ahead 1024 sectors Title: Doc No.: T1000 EPAP A - MO007185 - T1000 EPAP A - Dual Disk Replacement from 120GB to 250GB or 500GB MO007185 Revision: 1.0 Page 7 of 23

10

Tekelec 5200 Paramount Parkway Morrisville, NC 27560 888-FOR-TKLC

THIS DOCUMENT AND THE DATA DISCLOSED HEREIN OR HEREWITH IS PROPRIETARY AND IS NOT TO BE REPRODUCED, USED OR DISCLOSED IN WHOLE OR IN PART TO ANYONE WITHOUT THE WRITTEN PERMISSION OF TEKELEC. COPYRIGHT, TEKELEC 2007.

Event: 11 md12 : active raid1 hda12[0] hdc12[1] 204242240 blocks [2/2] [UU] md11 : active raid1 hda11[0] hdc11[1] 15880128 blocks [2/2] [UU] md10 : active raid1 hda10[0] hdc10[1] 4208896 blocks [2/2] [UU] md9 : active raid1 hda9[0] hdc9[1] 538048 blocks [2/2] [UU] md8 : active raid1 hda8[0] hdc8[1] 4208896 blocks [2/2] [UU] md7 : active raid1 hda7[0] hdc7[1] 538048 blocks [2/2] [UU] md6 : active raid1 hda6[0] hdc6[1] 1060160 blocks [2/2] [UU] md5 : active raid1 hda5[0] hdc5[1] 4208896 blocks [2/2] [UU] md3 : active raid1 hda3[0] hdc3[1] 2056256 blocks [2/2] [UU] md2 : active raid1 hda2[0] hdc2[1] 538112 blocks [2/2] [UU] md1 : active raid1 hda1[0] hdc1[1] 272960 blocks [2/2] [UU] unused devices: <none> 11 12 13 14 15 16 Verify all meta devices are in sync Execute failDisk script Verify output Shut down MPS-A Verify output Remove power from MPS A. Should not see recovery or resync. Each md should show active raid1 and [2/2] [UU]. # /usr/TKLC/plat/sbin/failDisk --splitOnly /dev/hdc Should see: Removal of /dev/hdc completed, OK. # init 0 Should see: Power down. If the breakers are clearly marked, power off the two corresponding breakers, otherwise remove the two power cables on the rear of MPS A. Title: Doc No.: T1000 EPAP A - MO007185 - T1000 EPAP A - Dual Disk Replacement from 120GB to 250GB or 500GB MO007185 Revision: 1.0 Page 8 of 23

Tekelec 5200 Paramount Parkway Morrisville, NC 27560 888-FOR-TKLC

THIS DOCUMENT AND THE DATA DISCLOSED HEREIN OR HEREWITH IS PROPRIETARY AND IS NOT TO BE REPRODUCED, USED OR DISCLOSED IN WHOLE OR IN PART TO ANYONE WITHOUT THE WRITTEN PERMISSION OF TEKELEC. COPYRIGHT, TEKELEC 2007.

17 18 19 20

Remove the /dev/hdc drive Set jumper on new drive to match original drive. Install new drive Disconnect /dev/hda

Hard drive C is on the right side of the lid. (see T1000 Hardware Manual) Note that youre trying to match (connect mode) not actual pins. Thus, comparison of the jumper diagrams will be necessary. Reconnect cables. Disconnect the ribbon and power cables from the good (/dev/hda) drive - hard drive A is on the left side of the lid. The Purpose is to see that system if system is booting from HDC? If system boots from HDC drive, then we need to remove the MBR from this.

21 22

Close lid Apply serial number labels

Close and secure lid, and if necessary slide server into rack and secure mounting brackets. We should clearly mark the label on the new hard disks that we have installed. The Label should tell if the hard drive is HDA or HDC. Just in case there is not already a label on the removed disks, please apply label on them too. Either re-apply power at the breaker panel or restore the power cable connections on rear of MPS A. If system does not boot, skip to step 30. If the system boots from the hdc drive, proceed to next step to clear out the Master Boot Record and meta device information.

23 24

Restore power to MPS A. Determine if clearing of MBR is needed Log on to MPSA as root Verify output Clear the Master Boot Record and meta device information from hda. Reboot MPS-A Go to step 24 Remove power from MPS A. Reconnect /dev/hda Close lid Restore power to MPS A.

25 26 27

Should see: Last login: <date> from <hostname> # dd if=/dev/zero of=/dev/hdc bs=512 count=1

28 29 30 31 32 33

# init 6

If the breakers are clearly marked, power off the two corresponding breakers, otherwise remove the two power cables on the rear of MPS A. Reconnect the ribbon and power cables on the good (/dev/hda) drive - hard drive a is on the left side of the lid. Close and secure lid, and if necessary slide server into rack and secure mounting brackets. Either re-apply power at the breaker panel or restore the power cable connections on rear of MPS A. Title: Doc No.: T1000 EPAP A - MO007185 - T1000 EPAP A - Dual Disk Replacement from 120GB to 250GB or 500GB MO007185 Revision: 1.0 Page 9 of 23

Tekelec 5200 Paramount Parkway Morrisville, NC 27560 888-FOR-TKLC

THIS DOCUMENT AND THE DATA DISCLOSED HEREIN OR HEREWITH IS PROPRIETARY AND IS NOT TO BE REPRODUCED, USED OR DISCLOSED IN WHOLE OR IN PART TO ANYONE WITHOUT THE WRITTEN PERMISSION OF TEKELEC. COPYRIGHT, TEKELEC 2007.

34 35 36 37 38 39 40 41

Log on to MPSA as root Verify output Change runlevel to 3 in order to speed resync Verify output Verify runlevel Verify output Partition new disk Verify output Should see: Last login: <date> from <hostname> # init 3 Should see: Messages of services stopping. Hit <Enter> to get prompt, and then: # runlevel Should see: 43 # /usr/TKLC/plat/sbin/cpDiskCfg hda hdc Should see: WARNING: Supplied disks do not have the same model number! Disk Geometry Check, OK! Partitioning of disk /dev/hdc completed, OK! # partprobe # cd /proc/sys/dev/raid/ # cat speed_limit_max If value is not 900,000 set it to 900,000: # echo 900000 > speed_limit_max

42 43 44

For safety, have the kernel reread the partition table Ensure maximum speed of resync is possible Speed should be 900000.

45 46 47

Re-mirror new disk Verify output Check progress of resync

# /usr/TKLC/plat/sbin/mdRepair background Note - Step takes several minutes. Should see: Critical repairs have been made, backgrounding as requested. # grep recovery\|resync /proc/mdstat Note - Keep waiting here unless mirror status is good. If there is any output, this would mean that Mirrors are not yet joined. We need to wait for some time. If you notice that mirroring is still progressing, then wait here or else go back to the step that was issued to join to mirrors. # syscheck

48 49

If any output go to step 45. When resync completes, perform system checks on MPS-A Verify no syscheck failures

50

The only failures should be in class proc module run due to the Pdba and Title: Doc No.: T1000 EPAP A - MO007185 - T1000 EPAP A - Dual Disk Replacement from 120GB to 250GB or 500GB MO007185 Revision: 1.0 Page 10 of 23

Tekelec 5200 Paramount Parkway Morrisville, NC 27560 888-FOR-TKLC

THIS DOCUMENT AND THE DATA DISCLOSED HEREIN OR HEREWITH IS PROPRIETARY AND IS NOT TO BE REPRODUCED, USED OR DISCLOSED IN WHOLE OR IN PART TO ANYONE WITHOUT THE WRITTEN PERMISSION OF TEKELEC. COPYRIGHT, TEKELEC 2007.

other than due to Pdba and Epap software being down. 51 Move Pdba_stopped and Epap_stopped executables back to Pdba and Epap Change runlevel to 4 in order to bring application software up Verify output Verify runlevel Verify output Perform system checks on MPS A Verify no syscheck failures Verify PDB, RTDB, and DSM statuses are good. Have customer restart provisioning and verify DB statuses. Done!

Epap software being stopped. There will also be errors in connecting to the maint process and using the alarm dispatch utility. # cd /etc/init.d # mv Epap_stopped Epap # mv Pdba_stopped Pdba # init 4

52

53 54 55 56 57 58 59 60

Should see: Messages of services starting. Hit <Enter> to get prompt, and then: # runlevel Should see: 34 # syscheck Should NOT get this output: One or more module in class "<class>" FAILED From GUI: use View RTDB Status, View PDBA Status, Run Syscheck, Capture Logs. From Eagle: use rept-stat-sccp, Rept-stat-db:display=all:db=mps, etc. Run uiEdit command on ACTIVE node to allow provisioning # uiEdit PDBA_REMOTE_PDBI_ALLOWED YES If needed, the next disk can be replaced the same night or at the next maintenance window.

HDA REPLACEMENT
1 2 Prepare system Enter BIOS Please execute PREPARE THE SYSTEM procedure Upon restarting the server, enter the BIOS setup utility by pressing the DEL key when the message Press DEL to run Setup (F4 on Remote keyboard) is displayed. If using a PS2 keyboard (preferred method) the DEL key must be pressed immediately after the 3 keyboard LEDs light during the boot process. NOTE: This message is only displayed for a few seconds and the DEL or (F4 on Remote keyboard) key must be pressed before the message disappears from Title: Doc No.: T1000 EPAP A - MO007185 - T1000 EPAP A - Dual Disk Replacement from 120GB to 250GB or 500GB MO007185 Revision: 1.0 Page 11 of 23

Tekelec 5200 Paramount Parkway Morrisville, NC 27560 888-FOR-TKLC

THIS DOCUMENT AND THE DATA DISCLOSED HEREIN OR HEREWITH IS PROPRIETARY AND IS NOT TO BE REPRODUCED, USED OR DISCLOSED IN WHOLE OR IN PART TO ANYONE WITHOUT THE WRITTEN PERMISSION OF TEKELEC. COPYRIGHT, TEKELEC 2007.

the screen or the boot sequence will continue as usual. The BIOS setup utility provides navigational tips which are displayed in the lower right hand corner of the screen, but in general moving around within the utility is accomplished using the right, left, up, and down arrow keys. To select the highlighted option, simply press the Enter key, and to return to the previous screen press the ESC key. If at anytime your output is garbled, in SecureCRT select Reset from the Edit menu. 3 Select Hard Disk Drives in the Boot Settings screen. Once inside the BIOS setup utility verify that the disk drive that is not being replaced is set to be the boot disk. The setting of interest is on the Boot settings screen within the BIOS setup utility. On the Boot settings screen select Hard Disk Drives as shown below. BIOS Boot Settings Screen

Verify that hdc is listed as the 1st drive.

After selecting Hard Disk Drives, modify the drive order (the order of the entries can be reversed by pressing either the + or - key) to make the SM drive (hdc) the 1st Drive. NOTE: The device names listed on the Hard Disk Drives screen are the model number of the servers disk drives with one entry prepended by PM- (Primary Master) HDA and the other by SM- (Secondary Master) HDC. BIOS Hard Disk Drives Screen

Title: Doc No.:

T1000 EPAP A - MO007185 - T1000 EPAP A - Dual Disk Replacement from 120GB to 250GB or 500GB MO007185 Revision: 1.0 Page 12 of 23

Tekelec 5200 Paramount Parkway Morrisville, NC 27560 888-FOR-TKLC

THIS DOCUMENT AND THE DATA DISCLOSED HEREIN OR HEREWITH IS PROPRIETARY AND IS NOT TO BE REPRODUCED, USED OR DISCLOSED IN WHOLE OR IN PART TO ANYONE WITHOUT THE WRITTEN PERMISSION OF TEKELEC. COPYRIGHT, TEKELEC 2007.

Save any changes

If this setting requires modification make the necessary change, (the order of the entries can be reversed by pressing either the + or - key), and then save the CMOS data by pressing F10, and then selecting OK when the confirmation dialog is presented. If the order shown is correct, simply press the ESC key or select Discard Changes and Exit on the main Exit screen and select OK when asked to confirm the exit.

After exiting the BIOS setup utility, the server will restart. Once the login prompt is presented, log in as the root user. Verify output Perform system checks on MPS-A Verify no syscheck failures other than due to Pdba and Epap software being down. Verify mirror status is good

Login: root Password: <Enter root password>

7 8 9

Should see: Last login: <date> from <hostname> # syscheck The only failures should be in class proc module run due to the Pdba and Epap software being stopped. There will also be errors in connecting to the maint process and using the alarm dispatch utility. # cat /proc/mdstat Personalities : [raid1] read_ahead 1024 sectors Title: Doc No.: T1000 EPAP A - MO007185 - T1000 EPAP A - Dual Disk Replacement from 120GB to 250GB or 500GB MO007185 Revision: 1.0 Page 13 of 23

10

Tekelec 5200 Paramount Parkway Morrisville, NC 27560 888-FOR-TKLC

THIS DOCUMENT AND THE DATA DISCLOSED HEREIN OR HEREWITH IS PROPRIETARY AND IS NOT TO BE REPRODUCED, USED OR DISCLOSED IN WHOLE OR IN PART TO ANYONE WITHOUT THE WRITTEN PERMISSION OF TEKELEC. COPYRIGHT, TEKELEC 2007.

Event: 11 md12 : active raid1 hda12[0] hdc12[1] 204242240 blocks [2/2] [UU] md11 : active raid1 hda11[0] hdc11[1] 15880128 blocks [2/2] [UU] md10 : active raid1 hda10[0] hdc10[1] 4208896 blocks [2/2] [UU] md9 : active raid1 hda9[0] hdc9[1] 538048 blocks [2/2] [UU] md8 : active raid1 hda8[0] hdc8[1] 4208896 blocks [2/2] [UU] md7 : active raid1 hda7[0] hdc7[1] 538048 blocks [2/2] [UU] md6 : active raid1 hda6[0] hdc6[1] 1060160 blocks [2/2] [UU] md5 : active raid1 hda5[0] hdc5[1] 4208896 blocks [2/2] [UU] md3 : active raid1 hda3[0] hdc3[1] 2056256 blocks [2/2] [UU] md2 : active raid1 hda2[0] hdc2[1] 538112 blocks [2/2] [UU] md1 : active raid1 hda1[0] hdc1[1] 272960 blocks [2/2] [UU] unused devices: <none> 11 12 13 14 15 16 Verify all meta devices are in sync Execute failDisk script Verify output Shut down MPS-A Verify output Remove power from MPS A. Should not see recovery or resync. Each md should show active raid1 and [2/2] [UU]. # /usr/TKLC/plat/sbin/failDisk splitOnly /dev/hda Should see: Removal of /dev/hda completed, OK. # init 0 Should see: Power down. If the breakers are clearly marked, power off the two corresponding breakers, otherwise remove the two power cables on the rear of MPS A. Title: Doc No.: T1000 EPAP A - MO007185 - T1000 EPAP A - Dual Disk Replacement from 120GB to 250GB or 500GB MO007185 Revision: 1.0 Page 14 of 23

Tekelec 5200 Paramount Parkway Morrisville, NC 27560 888-FOR-TKLC

THIS DOCUMENT AND THE DATA DISCLOSED HEREIN OR HEREWITH IS PROPRIETARY AND IS NOT TO BE REPRODUCED, USED OR DISCLOSED IN WHOLE OR IN PART TO ANYONE WITHOUT THE WRITTEN PERMISSION OF TEKELEC. COPYRIGHT, TEKELEC 2007.

17 18 19 20

Remove the /dev/hda drive Set jumper on new drive to match original drive. Install new drive Disconnect /dev/hdc

Hard drive A is on the left side of the lid. Note Note that youre trying to match (connect mode) not actual pins. Thus, comparison of the jumper diagrams will be necessary. Reconnect Cables. Disconnect the ribbon and power cables from the good (/dev/hdc) drive - hard drive C is on the right side of the lid. The Purpose is to see that system if system is booting from HDA? If system boots from HDA drive, then we need to remove the MBR from this. Close and secure lid, and if necessary slide server into rack and secure mounting brackets. We should clearly mark the label on the new hard disks that we have installed. The Label should tell if the hard drive is HDA or HDC. Just in case there is not already a label on the removed disks, please apply label on them too. Either re-apply power at the breaker panel or restore the power cable connections on rear of MPS A. If system does not boot, skip to step 30. If the system boots from the hda drive, proceed to next step to clear out the Master Boot Record and meta device information.

21 22

Close lid Apply serial number labels

23 24

Restore power to MPS A. Determine if clearing of MBR is needed Log on to MPSA as root Verify output Clear the Master Boot Record and meta device information from hda. Reboot MPS-A Go to step 24 Remove power from MPS A. Reconnect /dev/hdc Close lid Restore power to MPS A.

25 26 27

Should see: Last login: <date> from <hostname> # dd if=/dev/zero of=/dev/hda bs=512 count=1

28 29 30 31 32 33

# init 6

If the breakers are clearly marked, power off the two corresponding breakers, otherwise remove the two power cables on the rear of MPS A. Reconnect the ribbon and power cables on the good (/dev/hdc) drive - hard drive C is on the right side of the lid. Close and secure lid, and if necessary slide server into rack and secure mounting brackets. Either re-apply power at the breaker panel or restore the power cable connections on rear of MPS A. Title: Doc No.: T1000 EPAP A - MO007185 - T1000 EPAP A - Dual Disk Replacement from 120GB to 250GB or 500GB MO007185 Revision: 1.0 Page 15 of 23

Tekelec 5200 Paramount Parkway Morrisville, NC 27560 888-FOR-TKLC

THIS DOCUMENT AND THE DATA DISCLOSED HEREIN OR HEREWITH IS PROPRIETARY AND IS NOT TO BE REPRODUCED, USED OR DISCLOSED IN WHOLE OR IN PART TO ANYONE WITHOUT THE WRITTEN PERMISSION OF TEKELEC. COPYRIGHT, TEKELEC 2007.

34 35 36 37 38 39 40 41

Log on to MPSA as root Verify output Change runlevel to 3 in order to speed resync Verify output Verify runlevel Verify output Partition new disk Verify output Should see: Last login: <date> from <hostname> # init 3 Should see: Messages of services stopping. Hit <Enter> to get prompt, and then: # runlevel Should see: 43 # /usr/TKLC/plat/sbin/cpDiskCfg hdc hda Should see: Model number check, OK! Disk Geometry Check, OK! Partitioning of disk /dev/hda completed, OK! # partprobe # cd /proc/sys/dev/raid/ # cat speed_limit_max If value is not 900,000 set it to 900,000: # echo 900000 > speed_limit_max

42 43 44

For safety, have the kernel reread the partition table Ensure maximum speed of resync is possible Speed should be 900000.

45 46 47 48 49

Re-mirror new disk Verify output Check progress of resync If any output go to step 45 When resync completes, perform system checks on MPS-A Verify no syscheck failures other than due to Pdba and Epap software being down.

# /usr/TKLC/plat/sbin/mdRepair background Note - Step takes several minutes. Should see: Critical repairs have been made, backgrounding as requested. # grep recovery\|resync /proc/mdstat Note - Keep waiting here unless mirror status is good. If there is any output, this would mean that Mirrors are not yet joined. We need to wait for some time. If you notice that mirroring is still progressing, then wait here or else go back to the step that was issued to join to mirrors. # syscheck

50

The only failures should be in class proc module run due to the Pdba and Epap software being stopped. There will also be errors in connecting to the maint process and using the alarm dispatch utility. T1000 EPAP A - MO007185 - T1000 EPAP A - Dual Title: Disk Replacement from 120GB to 250GB or 500GB Doc No.: MO007185 Revision: 1.0 Page 16 of 23

Tekelec 5200 Paramount Parkway Morrisville, NC 27560 888-FOR-TKLC

THIS DOCUMENT AND THE DATA DISCLOSED HEREIN OR HEREWITH IS PROPRIETARY AND IS NOT TO BE REPRODUCED, USED OR DISCLOSED IN WHOLE OR IN PART TO ANYONE WITHOUT THE WRITTEN PERMISSION OF TEKELEC. COPYRIGHT, TEKELEC 2007.

51

Reboot server to have Onsite Technician change boot device to /dev/hda. Enter BIOS

# init 6 Note-This may have to be done several times before successful attempt to enter bios. Upon restarting the server, enter the BIOS setup utility by pressing the DEL key when the message Press DEL to run Setup (F4 on Remote keyboard) is displayed. If using a PS2 keyboard (preferred method) the DEL key must be pressed immediately after the 3 keyboard LEDs light during the boot process. NOTE: This message is only displayed for a few seconds and the DEL key must be pressed before the message disappears from the screen or the boot sequence will continue as usual. The BIOS setup utility provides navigational tips which are displayed in the lower right hand corner of the screen, but in general moving around within the utility is accomplished using the right, left, up, and down arrow keys. To select the highlighted option, simply press the Enter key, and to return to the previous screen press the ESC key. If at anytime your output is garbled, in SecureCRT select Reset from the Edit menu

52

53

Select Hard Disk Drives in the Boot Settings screen.

Once inside the BIOS setup utility verify that the HDA is set to be the boot disk. The setting of interest is on the Boot settings screen within the BIOS setup utility. On the Boot settings screen select Hard Disk Drives as shown below. BIOS Boot Settings Screen

Title: Doc No.:

T1000 EPAP A - MO007185 - T1000 EPAP A - Dual Disk Replacement from 120GB to 250GB or 500GB MO007185 Revision: 1.0 Page 17 of 23

Tekelec 5200 Paramount Parkway Morrisville, NC 27560 888-FOR-TKLC

THIS DOCUMENT AND THE DATA DISCLOSED HEREIN OR HEREWITH IS PROPRIETARY AND IS NOT TO BE REPRODUCED, USED OR DISCLOSED IN WHOLE OR IN PART TO ANYONE WITHOUT THE WRITTEN PERMISSION OF TEKELEC. COPYRIGHT, TEKELEC 2007.

54

Verify that hda is listed as the 1st drive.

After selecting Hard Disk Drives, modify the drive order (the order of the entries can be reversed by pressing either the + or - key) to make the PM drive (hda) the 1st Drive. NOTE: The device names listed on the Hard Disk Drives screen are the model number of the servers disk drives with one entry prepended by PM- (Primary Master) and the other by SM- (Secondary Master). BIOS Hard Disk Drives Screen

Title: Doc No.:

T1000 EPAP A - MO007185 - T1000 EPAP A - Dual Disk Replacement from 120GB to 250GB or 500GB MO007185 Revision: 1.0 Page 18 of 23

Tekelec 5200 Paramount Parkway Morrisville, NC 27560 888-FOR-TKLC

THIS DOCUMENT AND THE DATA DISCLOSED HEREIN OR HEREWITH IS PROPRIETARY AND IS NOT TO BE REPRODUCED, USED OR DISCLOSED IN WHOLE OR IN PART TO ANYONE WITHOUT THE WRITTEN PERMISSION OF TEKELEC. COPYRIGHT, TEKELEC 2007.

55

Save any changes

If this setting required modification make the necessary change, and then save the CMOS data by pressing F10, and then selecting OK when the confirmation dialog is presented. If the order shown is correct, simply press the ESC key or select Discard Changes and Exit on the main Exit screen and select OK when asked to confirm the exit. login: root Password: <Enter root password>

56

After exiting the BIOS setup utility, the server will restart. Once the login prompt is presented, log in as the root user. Verify output Move Pdba_stopped and Epap_stopped executables back to Pdba and Epap Start Epap software Verify output Start Pdba software Verify output Perform system checks on MPS A Verify no syscheck failures Verify PDB, RTDB, and DSM statuses are good. Have customer restart provisioning and verify DB statuses. Done!

57 58

Should see: Last login: <date> from <hostname> # cd /etc/init.d # mv Epap_stopped Epap # mv Pdba_stopped Pdba # ./Epap start Should see: EPAP application started. # ./Pdba start Should see: PDBA application started. # syscheck Should NOT get this output: One or more module in class "<class>" FAILED From GUI: use View RTDB Status, View PDBA Status, Run Syscheck, Capture Logs. From Eagle: use rept-stat-sccp, Rept-stat-db:display=all:db=mps, etc. Run uiEdit command on ACTIVE node to allow provisioning # uiEdit PDBA_REMOTE_PDBI_ALLOWED YES If needed, the next disk can be replaced at the same night or at the next maintenance window.

59 60 61 62 63 64 65 66 67

. Title: Doc No.: T1000 EPAP A - MO007185 - T1000 EPAP A - Dual Disk Replacement from 120GB to 250GB or 500GB MO007185 Revision: 1.0 Page 19 of 23

Tekelec 5200 Paramount Parkway Morrisville, NC 27560 888-FOR-TKLC

THIS DOCUMENT AND THE DATA DISCLOSED HEREIN OR HEREWITH IS PROPRIETARY AND IS NOT TO BE REPRODUCED, USED OR DISCLOSED IN WHOLE OR IN PART TO ANYONE WITHOUT THE WRITTEN PERMISSION OF TEKELEC. COPYRIGHT, TEKELEC 2007.

Backout Procedure
S T E P # 1 2 3 4 5 6 Steps to be completed. Shut down MPS-A Verify output Remove power from MPS A. Reinstall both original drives Power on MPS-A Enter BIOS This procedure outlines the steps for returning the original Hard Drives to service in a T1000 Server # init 0 Should see: Power down. If the breakers are clearly marked, power off the two corresponding breakers, otherwise remove the two power cables on the rear of MPS A.

Either re-apply power at the breaker panel or restore the power cable connections on rear of MPS A. Upon restarting the server, enter the BIOS setup utility by pressing the DEL key when the message Press DEL to run Setup (F4 on Remote keyboard) is displayed. NOTE: This message is only displayed for a few seconds and the DEL key must be pressed before the message disappears from the screen or the boot sequence will continue as usual. Title: Doc No.: T1000 EPAP A - MO007185 - T1000 EPAP A - Dual Disk Replacement from 120GB to 250GB or 500GB MO007185 Revision: 1.0 Page 20 of 23

Tekelec 5200 Paramount Parkway Morrisville, NC 27560 888-FOR-TKLC

THIS DOCUMENT AND THE DATA DISCLOSED HEREIN OR HEREWITH IS PROPRIETARY AND IS NOT TO BE REPRODUCED, USED OR DISCLOSED IN WHOLE OR IN PART TO ANYONE WITHOUT THE WRITTEN PERMISSION OF TEKELEC. COPYRIGHT, TEKELEC 2007.

Set /dev/hda to be the 1 st drive in the Hard Disk Drives section of the Boot Settings

If at anytime your output is garbled, in SecureCRT select Reset from the Edit menu After selecting Hard Disk Drives, modify the drive order (the order of the entries can be reversed by pressing either the + or - key) to make the PM drive (hda) the 1st Drive. NOTE: The device names listed on the Hard Disk Drives screen are the model number of the servers disk drives with one entry prepended by PM- (Primary Master) and the other by SM- (Secondary Master).

8 9 10 11 12 13 14

Log on as root Verify mirror status Repair mirrors Verify output Check progress of resync If any output go to step 10. Move Pdba_stopped and Epap_stopped executables back to Pdba and Epap Start Epap and Pdba Run syscheck and fix any problems Contact TAC or ITAC

# cat /proc/mdstat # /usr/TKLC/plat/sbin/mdRepair background Note - Step takes several minutes. Should see: Critical repairs have been made, backgrounding as requested. # grep recovery\|resync /proc/mdstat Note - Keep waiting here unless mirror status is good. If there is any output, this would mean that Mirrors are not yet joined. We need to wait for some time. If you notice that mirroring is still progressing, then wait here or else go back to the step that was issued to join to mirrors. # cd /etc/init.d # mv Epap_stopped Epap # mv Pdba_stopped Pdba # ./Epap start # ./Pdba start # syscheck Tekelec Call Center 1-888-367-8552

15 16 17

Title: Doc No.:

T1000 EPAP A - MO007185 - T1000 EPAP A - Dual Disk Replacement from 120GB to 250GB or 500GB MO007185 Revision: 1.0 Page 21 of 23

Tekelec 5200 Paramount Parkway Morrisville, NC 27560 888-FOR-TKLC

THIS DOCUMENT AND THE DATA DISCLOSED HEREIN OR HEREWITH IS PROPRIETARY AND IS NOT TO BE REPRODUCED, USED OR DISCLOSED IN WHOLE OR IN PART TO ANYONE WITHOUT THE WRITTEN PERMISSION OF TEKELEC. COPYRIGHT, TEKELEC 2007.

3. Customer Survey.
If you, our customer, have any comments that would improve the quality of our Method of Procedure, please send them along with your name, company name, site name and phone number to Tekelec at support@tekelec.com. Thank you for contributing to the quality of this document. Sincerely, Tekelec

Title: Doc No.:

T1000 EPAP A - MO007185 - T1000 EPAP A - Dual Disk Replacement from 120GB to 250GB or 500GB MO007185 Revision: 1.0 Page 22 of 23

Tekelec 5200 Paramount Parkway Morrisville, NC 27560 888-FOR-TKLC

THIS DOCUMENT AND THE DATA DISCLOSED HEREIN OR HEREWITH IS PROPRIETARY AND IS NOT TO BE REPRODUCED, USED OR DISCLOSED IN WHOLE OR IN PART TO ANYONE WITHOUT THE WRITTEN PERMISSION OF TEKELEC. COPYRIGHT, TEKELEC 2007.

3. APPENDIX A: CHANGES IN MOP AFTER WORK HAS STARTED

When a change in the MOP or in a step within this MOP is necessary, a concurrence must be obtained between Tekelec Customer Service and Tekelec Engineering prior to its implementation. Changes to this MOP will be reviewed, agreed upon and documented in the attached MOP Change Sheet TEKELEC METHOD OF PROCEDURE CHANGE RECORD
Page __1__ of __1__ Pages

List below all changes, including item # and reason for change. Change items must also be noted on the original MOP. Note: The Tekelec Technical Service Engineer approves any changes made on a detailed MOP. Note: Exceptions will require 3rd level review and acknowledgement before turning system back over to the customer. Author of Changes:
STEP-BY-STEP PROCEDURES

Date of Change:
RESPONSE Tekelec 3rd Level DATE

Custo mer*

Reviewed and approved by: Concurred in by TEKELEC:

Title

Date

Note: If applicable

Title: Doc No.:

T1000 EPAP A - MO007185 - T1000 EPAP A - Dual Disk Replacement from 120GB to 250GB or 500GB MO007185 Revision: 1.0 Page 23 of 23