You are on page 1of 71

Migrating to AIX 7.

2
with minimal downtime

Chris Gibson (cgibson@au1.ibm.com)


AIX & Power Systems Consultant
IBM Systems Lab Services

March 16th, 2018


AIX VUG
AIX VUG March 2018

• Chris Gibson (cgibson@au1.ibm.com)


IBM Lab Services AIX & Power Systems Consultant.

• IBM Redbooks co-author & technical editor/author for IBM Systems


Magazine and IBM developerWorks. “Chris’s AIX blog” on
developerWorks. IBM Champion for Power Systems.

• IBM Redooks: NIM A-Z, Exploiting WPARs and AIX 7.1 Differences
Guide.

• @cgibbo Twitter – All things AIX and POWER

© Copyright IBM Corporation 2018. Technical University/Symposia materials may not 2


IBM Systems Technical Events | ibm.com/training/events be reproduced in whole or in part without the prior written permission of IBM.
Minimize migration downtime with NIMADM
• Purpose
• This presentation demonstrates how to migrate to AIX 7.2 using the NIM Alternate Disk
Migration tool.

• You'll learn how to minimize the downtime required when migrating to the latest release
of the AIX operating system.

• NIM
• The AIX OS provides the Network Installation Manager (NIM).
• Assist in administering and updating large numbers of AIX systems.
• NIM Alternate Disk Migration (nimadm) feature.
• Perform AIX migrations without the need for lengthy outages.
• Demonstrate procedures we used to migrate our AIX systems.
• Assume you are already very familiar with AIX and NIM.
• Already have a NIM master in your environment.
• Assume you are NOT familiar with nimadm.
• Read this Redbook, NIM from A to Z in AIX 5L:
http://www.redbooks.ibm.com/redpieces/abstracts/sg247296.html?Open
© Copyright IBM Corporation 2018. Technical University/Symposia materials may not 3
IBM Systems Technical Events | ibm.com/training/events be reproduced in whole or in part without the prior written permission of IBM.
Migrating AIX Overview – conventional method
Without NIM(adm)…what would we do?

• Over the years, I've migrated to several new releases of the AIX OS.
• To do this I would have typically used one of the “old school” methods.
• These methods consisted of either :
A) Migration using the AIX installation media (CD, DVD, iso) or
B) Migration using NIM.
• Method A is still possible, even in virtualized environments via the use of File-Backed
devices.
• Method B is also perfectly viable by network booting the client LPAR and performing
the migration using a NIM master.

• http://gibsonnet.net/aix/AIX_Migration_with_File-Backed_VIOS_Devices.htm

• http://www.ibm.com/developerworks/aix/library/au-aix-system-migration-
installation/index.html
© Copyright IBM Corporation 2018. Technical University/Symposia materials may not 4
IBM Systems Technical Events | ibm.com/training/events be reproduced in whole or in part without the prior written permission of IBM.
Migrating AIX Overview – conventional method
• Virtual Media Library - File-backed devices - on VIOS. Poor man’s NIM.
$ lsrep
Size(mb) Free(mb) Parent Pool Parent Size Parent Free
25497 9979 rootvg 279552 132096

Name File Size Optical Access


AIX_v7.2_DVD_1.iso 3222 vtopt4 ro
AIX_v7.2_DVD_2.iso 3976 None ro

© Copyright IBM Corporation 2018. Technical University/Symposia materials may not 5


IBM Systems Technical Events | ibm.com/training/events be reproduced in whole or in part without the prior written permission of IBM.
Migrating AIX Overview - conventional method
• Boot the AIX system from either NIM or VML virtual CD. Perform migration.

© Copyright IBM Corporation 2018. Technical University/Symposia materials may not 6


IBM Systems Technical Events | ibm.com/training/events be reproduced in whole or in part without the prior written permission of IBM.
Migrating AIX Overview – conventional method
• BOS menus – select Migration Install, etc…and….wait.......................for install to finish. Fingers crossed!

© Copyright IBM Corporation 2018. Technical University/Symposia materials may not 7


IBM Systems Technical Events | ibm.com/training/events be reproduced in whole or in part without the prior written permission of IBM.
Conventional AIX migration methods = DOWNTIME!

• The downside with both of these methods is that they both require significant
downtime on the system while the migration takes place.

• This downtime could be anywhere from 30-45 minutes to an hour or more, depending
on the system.

• This can be a concern in environments with tight outage windows. Systems not
clustered for HA.

© Copyright IBM Corporation 2018. Technical University/Symposia materials may not 8


IBM Systems Technical Events | ibm.com/training/events be reproduced in whole or in part without the prior written permission of IBM.
Advantages
NIM Alternate Disk Migration
• Why should I use nimadm?

• The nimadm utility offers several advantages over a conventional migration.


• Creates a copy of a NIM client's rootvg (on a spare disk on the client).
• Similar to alternate disk install (alt_disk_copy).
• Migrate the alternate rootvg disk to a newer version of AIX.

• All of this can be done without disruption to the client .


• There is no outage required to perform the migration.

• After the migration is finished, the only downtime required will be a scheduled reboot of
the system.
• Another advantage is that the actual migration process occurs on the NIM master.
• Taking the load off the client LPAR.
• This reduces the processing overhead on the client.

• Minimizes the performance impact to the running applications.

© Copyright IBM Corporation 2018. Technical University/Symposia materials may not 9


IBM Systems Technical Events | ibm.com/training/events be reproduced in whole or in part without the prior written permission of IBM.
Advantages
NIM Alternate Disk Migration
• Benefits?

• For customers with a large number of AIX systems.


• nimadm tool supports migrating several clients at once.

• Benefits over other migration methods:

1. Reduced downtime for the client.


2. Migration is executed while the system is up and running.
3. No disruption to any of the applications or services running on the client.
4. Upgrade can be done at a time convenient to the administrator.
5. At a later stage, a reboot can be scheduled in order to restart the system at the
later level of AIX.
6. Back out, to a previous version of AIX, is simple. Does not require recovery from
mksysb.

© Copyright IBM Corporation 2018. Technical University/Symposia materials may not 10


IBM Systems Technical Events | ibm.com/training/events be reproduced in whole or in part without the prior written permission of IBM.
Advantages
NIM Alternate Disk Migration
• Other reasons to use nimadm.

• Process is very flexible: Can be customized – optional customization resources i.e.


image_data, bosinst_data, pre/post_migration scripts, exclude_files, and so on.

• Quick recovery from migration failures. All changes are performed on the rootvg copy
(altinst_rootvg).
• Any serious problems with the migration, the original rootvg is still available and the
system has not been impacted.

• If a migration fails or terminates at any stage, nimadm is able to quickly recover from the
event and clean up afterwards.
• There is little for the administrator to do except determine why the migration failed, rectify
the situation, and attempt the nimadm process again.

• If the migration completed but issues are discovered after the reboot, then the
administrator can back out easily by booting from the original rootvg disk.

© Copyright IBM Corporation 2018. Technical University/Symposia materials may not 11


IBM Systems Technical Events | ibm.com/training/events be reproduced in whole or in part without the prior written permission of IBM.
Preparing the NIM environment
for nimadm
• Preparation

• You must have a NIM master running AIX 7.2 or higher with the latest Technology Level
and Service Pack.

• The NIM master (as always) should be at the same or higher AIX level than the level
you are migrating to on the client.

• The lpp_source and SPOT NIM resources that have been selected for the migration
must match the AIX level to which you are migrating.

• The NIM master must have the bos.alt_disk_install.rte fileset installed in the AIX 7.2
SPOT that will be used for the migration.

• Your live applications & application data should NOT reside in or run from rootvg
i.e. DB2, Oracle, WAS, etc. Otherwise you need to stop applications before nimadm.
“…application servers and the database servers do not start normally after the nimadm
command operations are complete.”
• https://www.ibm.com/support/knowledgecenter/en/ssw_aix_72/com.ibm.aix.cmds4/nimadm.htm
© Copyright IBM Corporation 2018. Technical University/Symposia materials may not 12
IBM Systems Technical Events | ibm.com/training/events be reproduced in whole or in part without the prior written permission of IBM.
Preparing the NIM environment
for nimadm
• Preparation – continued.

• The target client must be registered with the NIM master as a standalone NIM client.

• The NIM master must be able to execute remote commands on the client using nimsh
or rsh.

• Ensure the NIM client has a spare disk (not allocated to a volume group) large enough
to contain a complete copy of its rootvg.
• If rootvg is mirrored, break the mirror and use one of the disks for the migration.
• Assign a new SAN disk for this purpose.

• Ensure the NIM master has a volume group (for example, nimadmvg) with enough free
space to cater for a complete copy of the client's rootvg.

• If more than one AIX migration is occurring for multiple NIM clients, make sure there is
capacity for a copy of each clients rootvg. Perhaps create a separate VG for each NIM
client that is migrating.

© Copyright IBM Corporation 2018. Technical University/Symposia materials may not 13


IBM Systems Technical Events | ibm.com/training/events be reproduced in whole or in part without the prior written permission of IBM.
Local disk caching versus NFS
nimadm
• Why not use NFS?

• By default, nimadm uses NFS for many of the tasks during the migration.
• This can be a problem on slower networks because NFS writes can be very expensive.

• To avoid using NFS, a Local Disk Caching option exists that can provide some
performance advantages.
• Local disk caching allows the NIM master to avoid having to use NFS to write to the
client.

• Local Disk Caching will create the client file systems in a volume group (of your choosing)
on the NIM master.

• It will then use streams (via nimsh or rshd) to cache all of the data from the client to the
file systems on the NIM master.

© Copyright IBM Corporation 2018. Technical University/Symposia materials may not 14


IBM Systems Technical Events | ibm.com/training/events be reproduced in whole or in part without the prior written permission of IBM.
Local disk caching - Advantages
nimadm
• Why Local Disk Caching?

• Local disk caching provides:


• Improved performance for nimadm operations on relatively slow networks.
• Improved performance for nimadm operations that are bottlenecked in NFS writes.
• Decreased CPU usage on the client.
• Client file systems not exported (security conscious).
• Allows TCB enabled systems to be migrated with nimadm*.
• For best performance, use a volume group on the NIM master that does not contain the
NIM resources being used for the AIX migration.

Some potential disadvantages:


• Cache file systems take up space on the NIM master.
• Increased CPU usage on the NIM master.
• Increased disk I/O activity on the master.
• Requires additional storage (when migrating lots of NIM clients).
• May need more CPU and memory on NIM master (when migrating lots of NIM clients).
*Starting with AIX 7.2, the Trusted Computing Base security model is not supported. Therefore, it is disabled during the traditional migration to AIX 7.2 and when you migrate to AIX 7.2 by
using the nimadm command.
© Copyright IBM Corporation 2018. Technical University/Symposia materials may not 15
IBM Systems Technical Events | ibm.com/training/events be reproduced in whole or in part without the prior written permission of IBM.
Local disk caching
nimadm

nimadm phases with disk caching


•There are 12 phases to nimadm, however…
•Here are the most important steps!

© Copyright IBM Corporation 2018. Technical University/Symposia materials may not 16


IBM Systems Technical Events | ibm.com/training/events be reproduced in whole or in part without the prior written permission of IBM.
Phases
nimadm
12 Phases
• The nimadm command performs a migration in 12 phases:

1. Master issues alt_disk_install to the client. Makes a copy of the clients rootvg to target disk.
In this phase, the alternate root volume group (altinst_rootvg) is created.

2. Master creates the cache file systems in the cache volume group (nimadmvg) on the NIM
master. Some initial checks for the required migration disk space are performed. VG name,
nimadmvg, is an example only. It can be named anything you wish (1 to 15 characters in
length).

3. The NIM master copies the NIM client's data to the cache file systems in nimadmvg. This
data copy is done via nimsh (or rsh).

4. If a pre-migration script resource has been specified, it is executed in this phase.

5. System configuration files are saved. Initial migration space is calculated and appropriate
file system expansions are made. The bos image is restored and the device database is
merged (similar to a conventional migration).

6. All system filesets are migrated using installp. Any required RPM images are also installed
during this phase.

© Copyright IBM Corporation 2018. Technical University/Symposia materials may not 17


IBM Systems Technical Events | ibm.com/training/events be reproduced in whole or in part without the prior written permission of IBM.
Phases
nimadm
Phases

7. If a post-migration script resource has been specified, it is executed in this phase.

8. The bosboot command is run to create a client boot image, which is written to the
client's alternate boot logical volume (alt_hd5).

9. All the migrated data is now copied from the NIM master's local cache file and
synced to the client's alternate rootvg via nimsh or rsh.

10. The NIM master cleans up and removes the local cache file systems.

11. The alt_disk_install command is called again to make the final adjustments and put
altinst_rootvg to sleep. The bootlist is set to the target disk.

12. Cleanup is performed to end the migration.

Note: Each phase can be executed individually by using the -P flag. The nimadm phases must be run
sequentially.
© Copyright IBM Corporation 2018. Technical University/Symposia materials may not 18
IBM Systems Technical Events | ibm.com/training/events be reproduced in whole or in part without the prior written permission of IBM.
Consider this…
nimadm
Before we start…
• A few points for you to consider.

• I recommended that you do not to make any changes to your system once the migration is
underway, such as adding local users, changing local passwords, adding print queues, and the
like. LDAP is fine.
• If possible, wait until the migration has finished and the system has been rebooted on the new
version of AIX.
• If you must perform administration tasks prior to the reboot, you should take note of the changes
and re-apply them to the system after it has been rebooted into AIX 7.2.
• We developed, tested, and verified our migration procedures several times before
implementing them on our production systems.
• This allowed us time to verify that the steps were correct and that the AIX migrations would
complete as expected. I recommend you do the same.
• If you have a multibos image in rootvg, remove it. AIX migrations are not supported with multibos
enabled systems.**
• Ensure all rootvg LVs are renamed to their legacy names. If necessary, create a new instance of
rootvg and reboot the LPAR.

**AIX 7.2 will allow migrations (i.e. 7.2 à NextAIX). You can now do preservation or migration operating
system installs on a multibos instance. Previously these wouldn’t be a recognised rootvg. NIMADM is
also acceptable. The logical volumes will be renamed to the normal lv names after the installation.
NIMADM: for this case, the rename doesn’t occur until you boot to the altinst_rootvg.
http://www.ibm.com/support/knowledgecenter/ssw_aix_72/com.ibm.aix.cmds3/multibos.htm
© Copyright IBM Corporation 2018. Technical University/Symposia materials may not 19
IBM Systems Technical Events | ibm.com/training/events be reproduced in whole or in part without the prior written permission of IBM.
Consider this…
nimadm
Before we start…
• Ensure all rootvg LVs are renamed to their legacy names. If necessary, create a new
multibos instance and reboot the LPAR. For example:
# multibos –sXp
# multibos –sX
# shutdown –Fr

• Remove the old multibos instance.


# multibos -R

• Confirm the legacy LV names are now in use i.e. not bos_.

© Copyright IBM Corporation 2018. Technical University/Symposia materials may not 20


IBM Systems Technical Events | ibm.com/training/events be reproduced in whole or in part without the prior written permission of IBM.
Let’s migrate…..

© Copyright IBM Corporation 2018. Technical University/Symposia materials may not 21


IBM Systems Technical Events | ibm.com/training/events be reproduced in whole or in part without the prior written permission of IBM.
Migrating to AIX 7.2 using nimadm
nimadm
Let’s start…
• Let's use nimadm now to migrate an AIX system.
• We will migrate a system from AIX 7.1 to AIX 7.2. The NIM master in this environment
is running AIX 7.2 TL0 SP1.
• Our NIM client is running AIX 7.1 TL3 SP3 (migrating to AIX 7.2 TL0 SP1).
• Ensure that you document the system and take a mksysb before performing any
maintenance activity. You know this already, right? But I have to say it!
• Ensure that you read the AIX 7.2 release notes and review the documented
requirements such as the amount of free disk space required.
• Check your firmware levels, HMC code, VIOS code etc. Use the FLRT to check
compatibility.

• Prior to a migration, it is always a good idea to run the pre_migration script on the
system to catch any issues (e.g. multibos) that may prevent the migration from
completing successfully. You can find this script on the AIX 7.2 installation media.

# loopmount -i $PWD/AIX_v7.2_Base_Install_DVD_1_of_2_TL_7200-01-00_112016.iso -o "-V cdrfs -o ro" -m /mnt

# cd /mnt ; find . | grep pre_mig


./usr/lpp/bos/pre_migration

© Copyright IBM Corporation 2018. Technical University/Symposia materials may not 22


IBM Systems Technical Events | ibm.com/training/events be reproduced in whole or in part without the prior written permission of IBM.
Migrating to AIX 7.2 using nimadm
nimadm
Pre-Migration checks…

• Run this script, review the output (in /home/pre_migration), and correct any issues
that it reports before migrating.

# ./pre_migration

All saved information can be found in: /home/pre_migration.120621143915


Checking size of boot logical volume (hd5).
Listing software that will be removed from the system.
Listing configuration files that will not be merged.
Listing configuration files that will be merged.
Saving configuration files that will be merged.
Running lppchk commands. This may take awhile.
Please check /home/pre_migration.120621143915/software_file_existence_check for possible errors.
Please check /home/pre_migration.120621143915/software_checksum_verification for possible errors.
Please check /home/pre_migration.120621143915/tcbck.output for possible errors.
All saved information can be found in: /home/pre_migration.120621143915
It is recommended that you create a bootable system backup of your system
before migrating.

• Commit any applied filesets: installp -c all.

© Copyright IBM Corporation 2018. Technical University/Symposia materials may not 23


IBM Systems Technical Events | ibm.com/training/events be reproduced in whole or in part without the prior written permission of IBM.
Migrating to AIX 7.2 using nimadm
nimadm
bos.alt_disk_install.rte

• The fileset bos.alt_disk_install.rte fileset is installed on the NIM master:

# lslpp -l bos.alt_disk_install.rte

Fileset Level State Description


---------------------------------------------------------------------------
Path: /usr/lib/objrepos
bos.alt_disk_install.rte 7.2.0.1 COMMITTED Alternate Disk Installation
Runtime
Path: /etc/objrepos
bos.alt_disk_install.rte 7.2.0.1 COMMITTED Alternate Disk Installation
Runtime

• And it is also installed in the AIX 7.2 TL0 SP1 SPOT:

# nim -o showres spotaix72TL0SP1 | grep bos.alt


bos.alt_disk_install.rte 7.2.0.1 C F Alternate Disk Installation

© Copyright IBM Corporation 2018. Technical University/Symposia materials may not 24


IBM Systems Technical Events | ibm.com/training/events be reproduced in whole or in part without the prior written permission of IBM.
Migrating to AIX 7.2 using nimadm
nimadm
Spare disk, nimsh (or rsh)…

• If rootvg is mirrored, break the mirror and reduce it to a single disk.


• Assign another SAN disk.
• This gives you a spare disk that can be used for the migration.
• To allow nimadm to do its job, we can use nimsh or temporarily enable rshd on the
client LPAR.
• We will disable it again after the migration.

# chsubserver -a -v shell -p tcp6 -r inetd


# refresh –s inetd
# cd /
# vi .rhosts
nim01 root
# chmod 600 .rhosts

• On the NIM master, I can now ‘nimsh’ (or 'rsh’) to the client and run a command as root.

# nim -o showlog AIXlpar1


# rsh AIXlpar1 whoami

© Copyright IBM Corporation 2018. Technical University/Symposia materials may not 25


IBM Systems Technical Events | ibm.com/training/events be reproduced in whole or in part without the prior written permission of IBM.
Migrating to AIX 7.2 using nimadm
nimadm
Ready to migrate…

• At this point I'm ready to migrate.


• The process will take around 30-45 minutes; all the while the applications on the
LPAR will continue to function as normal.
• On the NIM master, I have created a new volume group (VG) named nimadmvg.
• Note: Don’t use rendev to rename hdisk. Non-hdisk name will cause nimadm to fail.
Thanks to Shawn Bodily (PowerHAGuy) for the tip!
• This VG has enough capacity to cater for a full copy of the NIM clients rootvg.
• This VG will be empty until the migration is started.
• NIM client: spare disk with enough capacity for a full copy of its rootvg.
• On the master (nim1):
# mkvg –y nimadmvg hdisk9
# lsvg -l nimadmvg
nimadmvg:
LV NAME TYPE LPs PPs PVs LV STATE MOUNT POINT

• On the client (AIXlpar1):


# lspv
hdisk0 0000273ac30fdcfc rootvg active
hdisk1 000273ac30fdd6e None

© Copyright IBM Corporation 2018. Technical University/Symposia materials may not 26


IBM Systems Technical Events | ibm.com/training/events be reproduced in whole or in part without the prior written permission of IBM.
nimadm command line options
nimadm
Flags and options

• The nimadm command is executed from the NIM master.

# nimadm -j nimadmvg -c AIXlpar1 -s spotaix72 -l lpp_sourceaix72 -d hdisk1 –Y

Where:

• –j flag specifies the VG on the master which will be used for the migration
• -c is the client name
• -s is the SPOT name
• -l is the lpp_source name
• -d is the hdisk name for the alternate root volume group (altinst_rootvg)
• -Y agrees to the software license agreements for software that will be installed during the
migration.

© Copyright IBM Corporation 2018. Technical University/Symposia materials may not 27


IBM Systems Technical Events | ibm.com/training/events be reproduced in whole or in part without the prior written permission of IBM.
Running nimadm….
nimadm
Executing nimadm…
• Now I can sit back and watch the migration take place. Grab a tea or coffee!
• All migration activity is logged on the NIM master in the /var/adm/ras/alt_mig directory.
• For this migration, the log file name is AIXlpar1_alt_mig.log. Verbose –D –V.
• Here's a sample of some of the output you can expect to see for each phase:
MASTER DATE: Wed Sep 30 13:48:27 AEST 2015
CLIENT DATE: Wed Sep 30 13:48:27 AEST 2015
NIMADM PARAMETERS: -j nimvg -c AIXlpar1 -s spotaix72 -l lpp_sourceaix72 -d hdisk1 -Y
Starting Alternate Disk Migration.
+-----------------------------------------------------------------------------+
Executing nimadm phase 1.
+-----------------------------------------------------------------------------+
Cloning altinst_rootvg on client, Phase 1.
Client alt_disk_install command: alt_disk_copy -j -M 7.2 -P1 -d "hdisk1"
Calling mkszfile to create new /image.data file.
Checking disk sizes.
LOGICAL_VOLUME= hd11admin
FS_LV= /dev/hd11admin
Creating cloned rootvg volume group and associated logical volumes.
Creating logical volume alt_hd5
Creating logical volume alt_hd6
Creating logical volume alt_hd8
...etc...
Generating a list of files
for backup and restore into the alternate file system...
Phase 1 complete.

© Copyright IBM Corporation 2018. Technical University/Symposia materials may not 28


IBM Systems Technical Events | ibm.com/training/events be reproduced in whole or in part without the prior written permission of IBM.
Phase 1
nimadm
+-----------------------------------------------------------------------------+
Executing nimadm phase 1.
+-----------------------------------------------------------------------------+
Cloning altinst_rootvg on client, Phase 1.
Client alt_disk_install command: alt_disk_copy -j -M 7.2 -P1 -d "hdisk1"
Calling mkszfile to create new /image.data file.
Checking disk sizes.
LOGICAL_VOLUME= hd11admin
FS_LV= /dev/hd11admin
Creating cloned rootvg volume group and associated logical volumes.
Creating logical volume alt_hd5
Creating logical volume alt_hd6
Creating logical volume alt_hd8
Creating logical volume alt_hd4
Creating logical volume alt_hd2
Creating logical volume alt_hd9var
Creating logical volume alt_hd3
Creating logical volume alt_hd1
Creating logical volume alt_hd10opt
Creating logical volume alt_hd11admin
Creating logical volume alt_lg_dumplv
Creating logical volume alt_livedump
Creating /alt_inst/ file system.
Creating /alt_inst/admin file system.
Creating /alt_inst/home file system.
Creating /alt_inst/opt file system.
Creating /alt_inst/tmp file system.
Creating /alt_inst/usr file system.
Creating /alt_inst/var file system.
Creating /alt_inst/var/adm/ras/livedump file system.
Generating a list of files
for backup and restore into the alternate file system...
Phase 1 complete. © Copyright IBM Corporation 2018. Technical University/Symposia materials may not 29
IBM Systems Technical Events | ibm.com/training/events be reproduced in whole or in part without the prior written permission of IBM.
Phase 2, 3 & 4
nimadm
+-----------------------------------------------------------------------------+
Executing nimadm phase 2.
+-----------------------------------------------------------------------------+
Creating nimadm cache file systems on volume group nimadmvg.
Checking for initial required migration space.
Creating cache file system /AIXlpar1_alt/alt_inst
Creating cache file system /AIXlpar1_alt/alt_inst/admin
Creating cache file system /AIXlpar1_alt/alt_inst/home
Creating cache file system /AIXlpar1_alt/alt_inst/opt
Creating cache file system /AIXlpar1_alt/alt_inst/tmp
Creating cache file system /AIXlpar1_alt/alt_inst/usr
Creating cache file system /AIXlpar1_alt/alt_inst/var
Creating cache file system /AIXlpar1_alt/alt_inst/var/adm/ras/livedump

+-----------------------------------------------------------------------------+
Executing nimadm phase 3.
+-----------------------------------------------------------------------------+
Syncing client data to cache ...

+-----------------------------------------------------------------------------+
Executing nimadm phase 4.
+-----------------------------------------------------------------------------+
nimadm: There is no user customization script specified for this phase.

© Copyright IBM Corporation 2018. Technical University/Symposia materials may not 30


IBM Systems Technical Events | ibm.com/training/events be reproduced in whole or in part without the prior written permission of IBM.
Phase 5
nimadm
+-----------------------------------------------------------------------------+
Executing nimadm phase 5.
+-----------------------------------------------------------------------------+
Saving system configuration files.
Checking for initial required migration space.
Setting up for base operating system restore.
/AIXlpar1_alt/alt_inst
Restoring base operating system.
Merging system configuration files.
Running migration merge method: ODM_merge Config_Rules.
Running migration merge method: ODM_merge SRCextmeth.
Running migration merge method: ODM_merge SRCsubsys.
Running migration merge method: ODM_merge SWservAt.
Running migration merge method: ODM_merge pse.conf.
Running migration merge method: privcmds_mig.
Running migration merge method: privdevs_mig.
Running migration merge method: ODM_merge vfs.
Running migration merge method: ODM_merge xtiso.conf.
Running migration merge method: ODM_merge PdAtXtd.
Running migration merge method: ODM_merge PdDv.
Running migration merge method: convert_errnotify.
Running migration merge method: passwd_mig.
Running migration merge method: login_mig.
Running migration merge method: user_mrg.
Running migration merge method: secur_mig.
Running migration merge method: RoleMerge.
Running migration merge method: methods_mig.
Running migration merge method: mkusr_mig.

Running migration merge method: ODM_merge GAI.
Running migration merge method: ODM_merge PdAt.
Running migration merge method: merge_smit_db.
Running migration merge method: ODM_merge fix.
Running migration merge method: merge_swvpds.
Running migration merge method: SysckMerge.

© Copyright IBM Corporation 2018. Technical University/Symposia materials may not 31


IBM Systems Technical Events | ibm.com/training/events be reproduced in whole or in part without the prior written permission of IBM.
Phase 6
nimadm
+-----------------------------------------------------------------------------+
Executing nimadm phase 6.
+-----------------------------------------------------------------------------+
Installing and migrating software.
Checking space requirements for installp install.
Expanding /AIXlpar1_alt/alt_inst/ local filesystem.
Filesystem size changed to 1048576
Expanding /AIXlpar1_alt/alt_inst/usr local filesystem.
Filesystem size changed to 4653056
Expanding /AIXlpar1_alt/alt_inst/var local filesystem.
Filesystem size changed to 851968
Installing software with the installp installer.
+-----------------------------------------------------------------------------+
Pre-installation Verification...
+-----------------------------------------------------------------------------+
Verifying selections...done
Verifying requisites...done
Results...
…etc…
Filesets processed: 1 of 681 (Total time: 3 secs).

installp: APPLYING software for:


wio.vscsi 7.2.0.0

…etc…
install_all_updates: Checking for recommended maintenance level 7200-00.
install_all_updates: Executing /usr/bin/oslevel -rf, Result = 7200-00
install_all_updates: Verification completed.
install_all_updates: Log file is /var/adm/ras/install_all_updates.log
install_all_updates: Result = SUCCESS
Restoring device ODM database.

© Copyright IBM Corporation 2018. Technical University/Symposia materials may not 32


IBM Systems Technical Events | ibm.com/training/events be reproduced in whole or in part without the prior written permission of IBM.
Phase 7, 8 & 9
nimadm
+-----------------------------------------------------------------------------+
Executing nimadm phase 7.
+-----------------------------------------------------------------------------+
nimadm: There is no user customization script specified for this phase.

+-----------------------------------------------------------------------------+
Executing nimadm phase 8.
+-----------------------------------------------------------------------------+
Creating client boot image.
bosboot: Boot image is 53248 512 byte blocks.
Writing boot image to client's alternate boot disk hdisk1.

+-----------------------------------------------------------------------------+
Executing nimadm phase 9.
+-----------------------------------------------------------------------------+
Adjusting client file system sizes ...
Adjusting size for /
Adjusting size for /admin
Adjusting size for /home
Adjusting size for /opt
Adjusting size for /tmp
Adjusting size for /usr
Adjusting size for /var
Adjusting size for /var/adm/ras/livedump
Syncing cache data to client ...

© Copyright IBM Corporation 2018. Technical University/Symposia materials may not 33


IBM Systems Technical Events | ibm.com/training/events be reproduced in whole or in part without the prior written permission of IBM.
Phase 10
nimadm
+-----------------------------------------------------------------------------+
Executing nimadm phase 10.
+-----------------------------------------------------------------------------+
Unmounting client mounts on the NIM master.
forced unmount of /AIXlpar1_alt/alt_inst/var/adm/ras/livedump
forced unmount of /AIXlpar1_alt/alt_inst/var/adm/ras/livedump
forced unmount of /AIXlpar1_alt/alt_inst/var
forced unmount of /AIXlpar1_alt/alt_inst/var
forced unmount of /AIXlpar1_alt/alt_inst/usr
forced unmount of /AIXlpar1_alt/alt_inst/usr
forced unmount of /AIXlpar1_alt/alt_inst/tmp
forced unmount of /AIXlpar1_alt/alt_inst/tmp
forced unmount of /AIXlpar1_alt/alt_inst/opt
forced unmount of /AIXlpar1_alt/alt_inst/opt
forced unmount of /AIXlpar1_alt/alt_inst/home
forced unmount of /AIXlpar1_alt/alt_inst/home
forced unmount of /AIXlpar1_alt/alt_inst/admin
forced unmount of /AIXlpar1_alt/alt_inst/admin
forced unmount of /AIXlpar1_alt/alt_inst
forced unmount of /AIXlpar1_alt/alt_inst
Removing nimadm cache file systems.
Removing cache file system /AIXlpar1_alt/alt_inst
Removing cache file system /AIXlpar1_alt/alt_inst/admin
Removing cache file system /AIXlpar1_alt/alt_inst/home
Removing cache file system /AIXlpar1_alt/alt_inst/opt
Removing cache file system /AIXlpar1_alt/alt_inst/tmp
Removing cache file system /AIXlpar1_alt/alt_inst/usr
Removing cache file system /AIXlpar1_alt/alt_inst/var
Removing cache file system /AIXlpar1_alt/alt_inst/var/adm/ras/livedump
© Copyright IBM Corporation 2018. Technical University/Symposia materials may not 34
IBM Systems Technical Events | ibm.com/training/events be reproduced in whole or in part without the prior written permission of IBM.
Phase 11 & 12
nimadm
+-----------------------------------------------------------------------------+
Executing nimadm phase 11.
+-----------------------------------------------------------------------------+
Cloning altinst_rootvg on client, Phase 3.
Client alt_disk_install command: alt_disk_copy -j -M 7.2 -P3 -d "hdisk1"
## Phase 3 ###################
Verifying altinst_rootvg...
Modifying ODM on cloned disk.
forced unmount of /alt_inst/var/adm/ras/livedump
forced unmount of /alt_inst/var/adm/ras/livedump
forced unmount of /alt_inst/var
forced unmount of /alt_inst/var
forced unmount of /alt_inst/usr
forced unmount of /alt_inst/usr
forced unmount of /alt_inst/tmp
forced unmount of /alt_inst/tmp
forced unmount of /alt_inst/opt
forced unmount of /alt_inst/opt
forced unmount of /alt_inst/home
forced unmount of /alt_inst/home
forced unmount of /alt_inst/admin
forced unmount of /alt_inst/admin
forced unmount of /alt_inst
forced unmount of /alt_inst
Changing logical volume names in volume group descriptor area.
Fixing LV control blocks...
Fixing file system superblocks...
Bootlist is set to the boot disk: hdisk1 blv=hd5

+-----------------------------------------------------------------------------+
Executing nimadm phase 12.
+-----------------------------------------------------------------------------+
Cleaning up alt_disk_migration on the NIM master.
Cleaning up alt_disk_migration on client AIXlpar1.

© Copyright IBM Corporation 2018. Technical University/Symposia materials may not 35


IBM Systems Technical Events | ibm.com/training/events be reproduced in whole or in part without the prior written permission of IBM.
After the migration
nimadm
Migration complete!

• After the migration is complete, confirm that the bootlist is set to the altinst_rootvg disk.

# lspv | grep rootvg


hdisk0 0000273ac30fdcfc rootvg active << AIX 7.1
hdisk1 000273ac30fdd6e altinst_rootvg active << AIX 7.2

# bootlist -m normal –o
hdisk1 blv=hd5 pathid=0
hdisk1 blv=hd5 pathid=1

• At an agreed time, reboot the LPAR and confirm that the system is now running AIX 7.2.
# oslevel –s
7100-03-03-1415

# shutdown –Fr

; system reboots here…takes a few minutes.

# oslevel –s
7200-00-01-1543

© Copyright IBM Corporation 2018. Technical University/Symposia materials may not 36


IBM Systems Technical Events | ibm.com/training/events be reproduced in whole or in part without the prior written permission of IBM.
After the migration
nimadm
Health checks and clean up

• At this point, perform some general AIX system health checks to ensure that the system
is configured and running as expected.

• https://www.ibm.com/developerworks/community/wikis/home?lang=en#!/wiki/Power+Syst
ems/page/DIY+AIX+System+Health+Checklist

• You may want to consider upgrading other software such as openssl, openssh, lsof, etc
at this stage.

• If enabled, the rsh daemon can now be disabled after the migration.
• # chsubserver -d -v shell -p tcp6 -r inetd
• # refresh –s inetd
• # cd /
• # rm .rhosts
• # ln -s /dev/null .rhosts

• There is also a post_migration script that you can run to verify the migration. You can
find this script in /usr/lpp/bos, after the migration.

© Copyright IBM Corporation 2018. Technical University/Symposia materials may not 37


IBM Systems Technical Events | ibm.com/training/events be reproduced in whole or in part without the prior written permission of IBM.
After the migration
nimadm
Post migration
• The post_migration command performs the following actions:

• Verify fileset version consistency.


• Check the installation list from before the migration, and inform the user of any filesets that might
still need migrating.
• Compare saved and merged configuration scripts and save the differences.

https://www.ibm.com/support/knowledgecenter/en/ssw_aix_72/com.ibm.aix.install/migration_scripts.htm

# /usr/lpp/bos/post_migration

Checking for software that was unchanged during the migration.

Comparing saved configuration files.

Running lppchk commands. This may take awhile.

All saved information can be found in: /home/post_migration.170801122811

[home/post_migration.170801122811 # ls -ltr
total 115232
-rw-r--r-- 1 root system 87783 Aug 01 12:28 software_installed_after_migration
-rw-r--r-- 1 root system 1276 Aug 01 12:28 filesets_to_be_updated
-rw-r--r-- 1 root system 16984 Aug 01 12:28 filesets_updated
-rw-r--r-- 1 root system 0 Aug 01 12:29 software_consistency
-rw-r--r-- 1 root system 58868353 Aug 01 12:29 compared_saved_configuration_files
-rw-r--r-- 1 root system 731 Aug 01 12:29 software_link_existence_check
-rw-r--r-- 1 root system 290 Aug 01 12:29 software_file_existence_check
-rw-r--r-- 1 root system 290 Aug 01 12:30 software_checksum_verification

© Copyright IBM Corporation 2018. Technical University/Symposia materials may not 38


IBM Systems Technical Events | ibm.com/training/events be reproduced in whole or in part without the prior written permission of IBM.
After the migration
nimadm
Post migration

• With the migration finished, the applications are started and the application support
team verify that everything is functioning as expected.

• Take a mksysb and document the system configuration after the migration.

• Once we are all satisfied that the migration has completed successfully, we then return
rootvg to a mirrored disk configuration (only if required).

# lspv | grep old_rootvg


hdisk0 000071da26fe3bd0 old_rootvg
# alt_rootvg_op -X old_rootvg
; If mirrored rootvg
# extendvg –f rootvg hdisk0
# mirrorvg rootvg hdisk0
# bosboot -a -d /dev/hdisk0
# bosboot -a -d /dev/hdisk1
# bootlist -m normal hdisk0 hdisk1
# bootlist -m normal -o hdisk0 blv=hd5 hdisk1 blv=hd5

© Copyright IBM Corporation 2018. Technical University/Symposia materials may not 39


IBM Systems Technical Events | ibm.com/training/events be reproduced in whole or in part without the prior written permission of IBM.
After the migration
nimadm
Uh oh! Back out!

• If there was an issue with the migration, easily back out to the previous release of AIX.
• Instead of re-mirroring rootvg (previous slide), we would change the boot list to point at
the previous rootvg disk (old_rootvg) and reboot the LPAR.

# lspv | grep old_rootvg


hdisk0 000071da26fe3bd0 old_rootvg

# bootlist -m normal hdisk0


# bootlist -m normal –o
hdisk0 blv=hd5

# shutdown –Fr

• This is much simpler and faster than restoring a mksysb image (via NIM, tape, or
DVD), as you would with a conventional migration method.

© Copyright IBM Corporation 2018. Technical University/Symposia materials may not 40


IBM Systems Technical Events | ibm.com/training/events be reproduced in whole or in part without the prior written permission of IBM.
After the migration
nimadm
No more lengthy after hours upgrades!

• By using nimadm, we were able to reduce the overall downtime required when
migrating our systems to AIX 7.2.

• We were also given a convenient way to back out a migration, had it been necessary to
do so.

• I hope this provides you with some ideas on how to best migrate your systems to AIX
7.2, when the time comes.

• http://www.ibm.com/developerworks/aix/library/au-
migrate_nimadm/?S_TACT=105AGY20&S_CMP=HP

© Copyright IBM Corporation 2018. Technical University/Symposia materials may not 41


IBM Systems Technical Events | ibm.com/training/events be reproduced in whole or in part without the prior written permission of IBM.
Other tips, tricks and best practices
for AIX migration and NIM…..

© Copyright IBM Corporation 2018. Technical University/Symposia materials may not 42


IBM Systems Technical Events | ibm.com/training/events be reproduced in whole or in part without the prior written permission of IBM.
nimadm – migrate mksysb image
nimadm
Alternative migration technique

• Migrate mksysb image from one version of AIX to another.

• Migrate AIX 6.1 mksysb image to AIX 7.1 or 7.2.

• Similar to nimadm – uses cache file systems (nimadmvg).

• Use the migrated mksysb image to install a new LPAR via NIM.

• Source LPAR running AIX 6.1 TL6 SP3. Took a mksysb of the source system to NIM
master.
root@aix61lpar / # oslevel -s
6100-06-03-1048

• NIM mksysb resource named cg-aix61.


root@nim1 / # lsnim -t mksysb | grep cg
cg-aix61 resources mksysb

© Copyright IBM Corporation 2018. Technical University/Symposia materials may not 43


IBM Systems Technical Events | ibm.com/training/events be reproduced in whole or in part without the prior written permission of IBM.
nimadm – migrate mksysb image
nimadm
Alternative to standard nimadm - continued

• AIX 7.1 lpp_source and SPOT ready and waiting on my NIM master.

root@nim1 / # lsnim -t lpp_source | grep aix7


aix7tl0sp2 resources lpp_source

root@nim1 / # lsnim -t spot | grep aix7


spotaix7tl0sp2 resources spot

© Copyright IBM Corporation 2018. Technical University/Symposia materials may not 44


IBM Systems Technical Events | ibm.com/training/events be reproduced in whole or in part without the prior written permission of IBM.
nimadm – migrate mksysb image
nimadm
Alternative to standard nimadm - continued

• To migrate the mksysb image from AIX 6.1 to AIX 7.1, I ran the following nimadm
command:

# nimadm -T cg-aix61 -O /export/mksysb/cg-aix71 -s spotaix7tl0sp2 -l


aix7tl0sp2 -j nimadmvg -Y -N cg-aix71

• The flags indicate the following:


• -T specifies the existing AIX 6.1 NIM mksysb resource.
• -O specifies the output location for the migrated mksysb resource.
• -s indicates the AIX 7.1 NIM SPOT resource for the migration.
• -l indicates the AIX 7.1 NIM lpp_source resource for the migration.
• -j identifies the volume group that will be used on the NIM master to create file systems.
• -Y Agress to required software license agreements.
• -N specifies the name of the new AIX 7.1 NIM mksysb resource to be created after
migration.

© Copyright IBM Corporation 2018. Technical University/Symposia materials may not 45


IBM Systems Technical Events | ibm.com/training/events be reproduced in whole or in part without the prior written permission of IBM.
nimadm – migrate mksysb image
continued
Migrate mksysb…continued

• Output.
# nimadm -T cg-aix61 -O /export/mksysb/cg-aix71 -s spotaix7tl0sp2 -l aix7tl0sp2 -j nimadmvg -Y -N cg-aix71

Initializing the NIM master.


Verifying alt_disk_migration eligibility.
Initializing log: /var/adm/ras/alt_mig/cg-aix61_alt_mig.log
Starting Alternate Disk Migration.

+-----------------------------------------------------------------------------+
Executing nimadm phase 1.
+-----------------------------------------------------------------------------+
Processing target mksysb resource cg-aix61.
Restoring /image.data from mksysb image.

+-----------------------------------------------------------------------------+
Executing nimadm phase 2.
+-----------------------------------------------------------------------------+
Creating nimadm cache file systems on volume group nimadmvg.
Checking for initial required migration space.
Creating cache file system /cg-aix61_mm_alt/alt_inst
Creating cache file system /cg-aix61_mm_alt/alt_inst/admin
Creating cache file system /cg-aix61_mm_alt/alt_inst/home
Creating cache file system /cg-aix61_mm_alt/alt_inst/opt
Creating cache file system /cg-aix61_mm_alt/alt_inst/tmp
Creating cache file system /cg-aix61_mm_alt/alt_inst/usr
Creating cache file system /cg-aix61_mm_alt/alt_inst/var
Creating cache file system /cg-aix61_mm_alt/alt_inst/var/adm/ras/livedump

© Copyright IBM Corporation 2018. Technical University/Symposia materials may not 46


IBM Systems Technical Events | ibm.com/training/events be reproduced in whole or in part without the prior written permission of IBM.
nimadm – migrate mksysb image
continued
Migrate mksysb…NIM mksysb resource - continued

• New NIM msysb resource after successful migration.


+-----------------------------------------------------------------------------+
Executing nimadm phase 11.
+-----------------------------------------------------------------------------+
Defining NIM mksysb resource ...
New NIMmksysb resource name is cg-aix71

• Once the migration had completed, I found a new mksysb file in /export/mksysb (as
expected). The file was named cg-aix71. And a new NIM mksysb resource was now
available.
root@nim1 / # ls -ltr /export/mksysb/ | grep aix7
-rw-r--r-- 1 root system 1967411200 Jan 06 11:00 cg-aix71

root@nim1 / # lsnim -t mksysb | grep aix7


cg-aix71 resources mksysb

© Copyright IBM Corporation 2018. Technical University/Symposia materials may not 47


IBM Systems Technical Events | ibm.com/training/events be reproduced in whole or in part without the prior written permission of IBM.
nimadm – migrate mksysb image
continued
Migrate mksysb…NIM mksysb resource - continued

• I was able to use the new, migrated, AIX 7.1 mksysb image to install a new LPAR.
AIX Version 7
Copyright IBM Corporation, 1982, 2010.
login: root
root's Password:
*******************************************************************************
* *
* Welcome to AIX Version 7.1! *
* *
* *
* Please see the README file in /usr/lpp/bos for information pertinent to *
* this release of the AIX Operating System. *
* *
*******************************************************************************
Last login: Thu Jan 6 13:21:04 EETDT 2011 on /dev/vty0

# oslevel -s
7100-00-02-1041

https://www.ibm.com/developerworks/mydeveloperworks/blogs/cgaix/entry/migrating_to_aix_7_1_with_nimadm?lang=en

© Copyright IBM Corporation 2018. Technical University/Symposia materials may not 48


IBM Systems Technical Events | ibm.com/training/events be reproduced in whole or in part without the prior written permission of IBM.
Other tips and tricks
Migrating to AIX 7.2 – JFS to JFS2
Other considerations – continued…

• nimadm does NOT convert rootvg file systems from JFS to JFS2.

• I have requested this feature. You should too!


https://www.ibm.com/developerworks/rfe/execute?use_case=viewRfe&CR_ID=117376

• Starting with 6.1 TL4, alt_disk_copy has new –T flag to convert to JFS2.

• One option is to use nimadm to migrate, then run alt_disk_copy with –T.

• Really only a problem for AIX 5.3 systems – alt_disk_copy –T not available.

• If you are on AIX 6.1 TL4 or higher, run alt_disk_copy –T first, then migrate to AIX 7.2.

https://www.ibm.com/developerworks/mydeveloperworks/blogs/cgaix/entry/convert_rootvg_fil
e_systems_to_jfs2_using_alt_disk_copy?lang=en

© Copyright IBM Corporation 2018. Technical University/Symposia materials may not 49


IBM Systems Technical Events | ibm.com/training/events be reproduced in whole or in part without the prior written permission of IBM.
Other tips and tricks
nimadm – pre & post scripts
Other considerations – continued…

• Pre & Post migration scripts.

-a PreMigrationScript Specifies the pre-migration NIM script resource.


-z PostMigrationScript Specifies the post-migration NIM script resource.

• Use post migration script to update 3rd party MPIO device driver filesets

• e.g. HDLM, HDS ODM, etc.

# nimadm -j nimadmvg -c lparaix01 -s spotaix610605 -l


lpp_sourceaix610605 -d hdisk2 -z XYZPOST –Y

https://www.ibm.com/developerworks/mydeveloperworks/blogs/cgaix/entry/using_a
_post_migration_script_with_nimadm1?lang=en

© Copyright IBM Corporation 2018. Technical University/Symposia materials may not 50


IBM Systems Technical Events | ibm.com/training/events be reproduced in whole or in part without the prior written permission of IBM.
Other tips and tricks
Migrating to AIX 6.1 – from AIX 5.X – with a 7.2 NIM master
AIX 7.2 NIM master - migrating clients to from AIX 5.3 to 6.1 (6.1 is EOS April 2017)

• If you plan on using your AIX 7.2 NIM master to migrate your AIX 5.3 clients to AIX 6.1,
then make sure you install the AIX 7.2 bos.alt_disk_install.rte fileset into the AIX 6.1
SPOT resource first. Failure to do so will result in your nimadm operation reporting the
following error message:

# nimadm -j nimadmvg -c lparaix53 -s spotaix610605 -l lpp_sourceaix610605 -d hdisk2 -Y


Initializing the NIM master.
Initializing NIM client lparaix53.
0042-001 nim: processing error encountered on "master":
/usr/bin/lslpp: Fileset bos.alt_disk_install.rte not installed.

0505-204 nimadm: SPOT spotaix610605 does not have bos.alt_disk_install.rte installed.


0505-205 nimadm: The level of bos.alt_disk_install.rte installed in SPOT
spotaix610605 (0.0.0.0) does not match the NIM master's level (7.2.0.1).
Cleaning up alt_disk_migration on the NIM master.

• You can verify the correct fileset is installed in your 6.1 SPOT using the following nim
command:

# nim -o showres spotaix610605 | grep bos.alt_disk_install.rte


bos.alt_disk_install.rte 7.2.0.1 A F Alternate Disk Installation

• Applies to AIX 5.3 -> 7.1 and AIX 6.1 -> 7.1 also.
© Copyright IBM Corporation 2018. Technical University/Symposia materials may not 51
IBM Systems Technical Events | ibm.com/training/events be reproduced in whole or in part without the prior written permission of IBM.
Other tips and tricks
Migrating to AIX 6.1 – from AIX 5.X – with a 7.2 NIM master
AIX 7.2 NIM master - migrating clients to from AIX 5.3 to 6.1 - continued…

• You must install the AIX 7.2 bos.alt_disk_install.rte fileset into your AIX 6.1 SPOT
resource.

# smit nim_res_op

....etc...

> spotaix610605

Customize a SPOT

Type or select values in entry fields.


Press Enter AFTER making all desired changes.

[Entry Fields]
* Resource Name spotaix610605
* Source of Install Images [lpp_sourceaix72] +
Fileset Names [bos.alt_disk_install.rte]

© Copyright IBM Corporation 2018. Technical University/Symposia materials may not 52


IBM Systems Technical Events | ibm.com/training/events be reproduced in whole or in part without the prior written permission of IBM.
Other tips and tricks
nimadm & multibos tip
Multibos /bos_inst

• As mentioned earlier, multibos is not supported in nimadm environments. Before you


start a nimadm migration make sure you have removed any old standby BOS instance
and that your rootvg logical volumes are not using any bos_ LV names.
• Exception: AIX 7.2 will allow it. AIX 7.2 à NextAIX.

• During our tests we found that even though we removed the standby instance (multibos
–R), the nimadm process failed with the following error:
+-----------------------------------------------------------------------------+
Executing nimadm phase 11.
+-----------------------------------------------------------------------------+
Cloning altinst_rootvg on client, Phase 3.
Client alt_disk_install command: alt_disk_copy -j -M 7.1 -P3 -d "hdisk1"
## Phase 3 ###################
Verifying altinst_rootvg...
alt_disk_copy: 0505-218 ATTENTION: init_multibos() returned an unexpected result.
Cleaning up.
…etc..
0505-187 nimadm: Error cloning altinst_rootvg on client.
Cleaning up alt_disk_migration on the NIM master.
Cleaning up alt_disk_migration on client lpar1.
Client alt_disk_install command: alt_disk_install -M 7.1 -X
Bootlist is set to the boot disk: hdisk0 blv=hd5

• Unfortunately, it appears that ‘multibos –R’ may not clean up the /bos_inst
directory. If this directory exists the nimadm operation will most likely fail. The simple
fix (in our case) was to remove the /bos_inst directory before attempting the AIX
migration.
# rm –r /bos_inst
© Copyright IBM Corporation 2018. Technical University/Symposia materials may not 53
IBM Systems Technical Events | ibm.com/training/events be reproduced in whole or in part without the prior written permission of IBM.
Other tips and tricks
Migrating AIX 4.3.3 to AIX v7.x

Can I upgrade AIX 4.3.3. to AIX 5, 6 or 7, using nimadm? Yes, you can!
• A customer of mine actually did this….needed to test an old & unsupported
application on AIX 4.3.3 and then migrate to AIX 7.1.
• Installed AIX 4.3.3 (from CD) on an old R/S6000 F50! PowerPC 604e processor!
• Put it on his network and used nimadm to migrate it to AIX 5.3. It all worked. Took 5.3
mksysb & installed on POWER7 795. Migrated to 7.1.
• One problem in nimadm phase 8 with 4.3.3 to 5.3 migration…
• Because in v5.X the boot image requires 12 MB of space!
• 1 PP of 8MB in size is not sufficient which is why NIMADM failed.
• Extend the size of hd5 and away we go…
NIMADM PARAMETERS: -j nimadmvg -c uataixl03 -s spotaix531208 -l lpp_sourceaix531208 -d hdisk1 -Y -V

Executing nimadm phase 8.


+-----------------------------------------------------------------------------+
Creating client boot image.
bosboot: Boot image is 44452 512 byte blocks.
Writing boot image to client's alternate boot disk hdisk1.
dd: /alt_inst/usr/lib/boot/bootinfo.txt: A file or directory in the path name does not exist.
44452+0 records in.
44452+0 records out.
dd: 0511-053 The write failed.
: There is a request to a device or address that does not exist.
0505-184 nimadm: Error writing client boot image.
Cleaning up alt_disk_migration on the NIM master.
------------------------------------------------------------

Workaround…
# extendlv hd5 1
# bosboot -ad /dev/hdisk0
; re-run nimadm.
© Copyright IBM Corporation 2018. Technical University/Symposia materials may not 54
IBM Systems Technical Events | ibm.com/training/events be reproduced in whole or in part without the prior written permission of IBM.
Other tips and tricks
Migrating to AIX 7.2
Other considerations….

• Latest AIX 7.2 installation tips


http://www-01.ibm.com/support/docview.wss?uid=isg1SSRVPOAIX72TIPS170127-1543

• Preparing to Migrate in AIX


http://www-01.ibm.com/support/docview.wss?uid=isg3T1011431

• This document covers migration preparation information.


• Information regarding version 5, 6 , and 7 installation
• Recommendations directly prior to migration
• Making sure your system is ready to migrate
• Walk-thru of the Base System Install menus for version 5
• Walk-thru of the Base System Install menus for version 6 & 7
What migration will do to your system
• Post migration checks

© Copyright IBM Corporation 2018. Technical University/Symposia materials may not 55


IBM Systems Technical Events | ibm.com/training/events be reproduced in whole or in part without the prior written permission of IBM.
Other tips and tricks
Migrating to AIX 7.2
IBM SDDPCM and NIMADM
• Note: If your NIM master and/or clients use the SDDPCM multi-pathing disk driver (with IBM SVC or other IBM disk),
please don’t forget that you’ll need to update the SDDPCM fileset to the correct version for AIX 7.2.

• The devices.sddpcm.71.rte fileset will need to be removed and devices.sddpcm.72.rte installed. Boot from
SAN.

• IBM support has scripts to assist with this process when performing conventional migrations (i.e. from media and/or the
VML) and with NIM/nimadm.

• There are several different scripts, depending on the type of migration you are performing. For example, if you are
performing a conventional migration then you can request the “Procedures for migrating SDDPCM before AIX
Operating Systems migration or Virtual I/O Server migration” scripts (two scripts: migratePCMbeforeOSmigrate.sh and
restorelvm.sh).

• If you are migrating a NIM client, using nimadm, then should request the “Migrate SDDPCM along with the nimadm
method of AIX OS migration via NIM for SAN boot or Non-SAN boot client” scripts (four scripts:
preSDDPCMOSmigration.sh, postSDDPCMOSmigration.sh, savelvm.sh and restorelvm.sh).

• Contact IBM support for the nimadm pre/post migration scripts. Open a PMR.

Overview of SDDPCM migration methods with IBM AIX (July 2017)


https://www-01.ibm.com/common/ssi/cgi-bin/ssialias?htmlfid=TSW03553USEN&

Upgrading the AIX OS with SDD or SDDPCM installed or migrating SDD to SDDPCM
and booting AIX rootvg from a SAN device
http://www-01.ibm.com/support/docview.wss?uid=ssg1S1002845

© Copyright IBM Corporation 2018. Technical University/Symposia materials may not 56


IBM Systems Technical Events | ibm.com/training/events be reproduced in whole or in part without the prior written permission of IBM.
Other tips and tricks
Migrating to AIX 7.2
How do I migrate my NIM master to AIX 7.2?
• Do you have a VIOS on the same frame as your NIM master?
• Use the VIOS Virtual Media Library (VML). Conventional migration method using install media (ISO).

• Migrate Your NIM Master to AIX 7.2 Using the VIOS VML
• http://www.ibmsystemsmag.com/aix/administrator/systemsmanagement/migrate-NIM-master/

© Copyright IBM Corporation 2018. Technical University/Symposia materials may not 57


IBM Systems Technical Events | ibm.com/training/events be reproduced in whole or in part without the prior written permission of IBM.
Other tips and tricks
Migrating to AIX 7.2 - Tunables
AIX Tunables Post Migration

• After an AIX migration, I usually like to run the tuncheck command to verify the current
tunable parameters are valid. One area that may indicate a tuning problem is the AIX
error report. If you see the following messages in the errpt output, you may want to
verify the current settings are valid:

IDENTIFIER TIMESTAMP T C RESOURCE_NAME DESCRIPTION


D221BD55 0523115112 I O perftune RESTRICTED TUNABLES MODIFIED AT REBOOT
---------------------------------------------------------------------------
LABEL: TUNE_RESTRICTED
IDENTIFIER: D221BD55

Date/Time: Wed May 23 11:51:16 EET 2012


Sequence Number: 676
Machine Id: 00C342C64C00
Node Id: lparaix53
Class: O
Type: INFO
WPAR: Global
Resource Name: perftune

Description
RESTRICTED TUNABLES MODIFIED AT REBOOT

Probable Causes
SYSTEM TUNING

User Causes
TUNABLE PARAMETER OF TYPE RESTRICTED HAS BEEN MODIFIED

Recommended Actions
REVIEW TUNABLE LISTS IN DETAILED DATA
Detail Data
LIST OF TUNABLE COMMANDS CONTROLLING MODIFIED RESTRICTED TUNABLES AT REBOOT, SEE FILE /etc/tunables/lastboot.log
vmo

© Copyright IBM Corporation 2018. Technical University/Symposia materials may not 58


IBM Systems Technical Events | ibm.com/training/events be reproduced in whole or in part without the prior written permission of IBM.
Other tips and tricks
Migrating to AIX 7.2 - Tunables
AIX Tunables Post Migration - continued

• In the output above you’ll notice that we are advised to check the
/etc/tunables/lastboot.log for a modified restricted vmo tuning parameter.
• At this point I usually like to run the tuncheck command against the current
/etc/tunables/nextboot file and review its output.

• As you can see, in the example below, we are warned that several restricted tunables are
not set to their default values.
• These values may not be appropriate for your newly migrated AIX 7.2 (or 7.1 or 6.1)
system.

• Settings that worked well with 5.3 are most likely no longer appropriate with 7.2.

# tuncheck -p -f /etc/tunables/nextboot
Warning: restricted tunable lrubucket is not at default value
Warning: restricted tunable strict_maxperm is not at default value
Warning: unknown parameter lru_file_repage in stanza vmo
Warning: restricted tunable maxperm% is not at default value
Warning: restricted tunable maxclient% is not at default value
Checking successful

© Copyright IBM Corporation 2018. Technical University/Symposia materials may not 59


IBM Systems Technical Events | ibm.com/training/events be reproduced in whole or in part without the prior written permission of IBM.
Other tips and tricks
Migrating to AIX 7.2 - Tunables
AIX Tunables Post Migration - continued

• Based on the output above, the tuning for this newly migrated 7.2 system appears to be
inappropriate. Unless we have a valid reason (which has been verified by IBM AIX
support) we should set these tunables to their default AIX 7.2 settings.

• You can reset individual tunables to their defaults using the –d flag and the
corresponding tuning command. For example to set the maxperm% tunable to its
default you would run the following vmo command:

# vmo -p -d maxperm%
Modification to restricted tunable maxperm%, confirmation required yes/no yes
Setting maxperm% to 90 in nextboot file
Setting maxperm% to 90
Warning: a restricted tunable has been modified

• If you want to set all the vmo tunables back to their defaults you would run the following
vmo command with the –D option (requires bosboot & reboot):

# vmo -r -D

© Copyright IBM Corporation 2018. Technical University/Symposia materials may not 60


IBM Systems Technical Events | ibm.com/training/events be reproduced in whole or in part without the prior written permission of IBM.
Other tips and tricks
Migrating to AIX 7.2 - Tunables
AIX Tunables Post Migration - continued

• Unless you’ve permanently set restricted tunables in your /etc/tunables/nexboot file,


the migration will change the systems default tuning to match the newer version of AIX.
• For example, we observed the following tuning changes on our AIX 5.3 system after
migrating to 7.2.

• The maxperm default value changed from 80 to 90:

maxperm% 80 80 80 1 100 % memory D


maxperm% 90 90 90 1 100 % memory D

• The minperm default value changed from 20 to 3:

minperm% 20 20 20 1 100 % memory D


minperm% 3 3 3 1 100 % memory D

• Note that starting with AIX 7.1 lru_file_repage is hardcoded to 0 and removed from the
list of vmo tunables. Please refer to the following document, Oracle Architecture and
Tuning on AIX v2.20, for more information. http://www-
03.ibm.com/support/techdocs/atsmastr.nsf/WebIndex/WP100883

© Copyright IBM Corporation 2018. Technical University/Symposia materials may not 61


IBM Systems Technical Events | ibm.com/training/events be reproduced in whole or in part without the prior written permission of IBM.
After the migration
alt_rootvg_op tip
What’s on the old rootvg disk?

• Wake up the old rootvg and use chroot to find out!


# oslevel –s << Booted on AIX 7.2
7200-01-02-1717

# alt_rootvg_op -W -d hdisk0 << Wake up the old rootvg


Waking up old_rootvg volume group ...

# chroot /alt_inst /alt_inst/bin/ksh oslevel –s << Check oslevel level with chroot on old rootvg (/alt_inst)
7100-04-03-1642 << It’s AIX 7.1 TL4 SP3

# alt_rootvg_op -S -d hdisk0 << Put old rootvg to sleep again. YOU MUST DO THIS!!!
Putting volume group altinst_rootvg to sleep ...
forced unmount of /alt_inst/var/adm/ras/livedump
forced unmount of /alt_inst/var/adm/ras/livedump
forced unmount of /alt_inst/var
forced unmount of /alt_inst/var
forced unmount of /alt_inst/usr
forced unmount of /alt_inst/usr
forced unmount of /alt_inst/tmp
forced unmount of /alt_inst/tmp
forced unmount of /alt_inst/opt
forced unmount of /alt_inst/opt
forced unmount of /alt_inst/home
forced unmount of /alt_inst/home
forced unmount of /alt_inst/admin
forced unmount of /alt_inst/admin
forced unmount of /alt_inst
forced unmount of /alt_inst
Fixing LV control blocks...
Fixing file system superblocks...
© Copyright IBM Corporation 2018. Technical University/Symposia materials may not 62
IBM Systems Technical Events | ibm.com/training/events be reproduced in whole or in part without the prior written permission of IBM.
What version of AIX is installed on that hdisk?
• Look for the pad string in the blv by running the following command:

# /usr/lpp/bosinst/blvset -d /dev/hdiskX –glevel

• The result should show your current version (3.2, 4.2, 4.3, 5.1, 5.2, 5.3, 6.1, 7.1 or 7.2,
you get the idea!). If it does not, run the following command:

# /usr/lpp/bosinst/blvset -d /dev/hdiskX –plevel

• It will move the cursor to the next line and wait for input. For the input of this command,
enter your version (e.g. 7.2), then press Enter.
• In an attempted migration from one AIX Version to a higher one, no migration option is
presented on BOS menu. Can be fixed with blvset (or bosboot –ad /dev/ipldevice).
http://www-01.ibm.com/support/docview.wss?uid=isg3T1025509

# lspv
hdisk0 00f94f58a3b21780 rootvg active
hdisk1 00f94f58a4464da1 None
# /usr/lpp/bosinst/blvset -d /dev/hdisk0 –glevel
7.2
# /usr/lpp/bosinst/blvset -d /dev/hdisk1 –glevel
7.1
© Copyright IBM Corporation 2018. Technical University/Symposia materials may not 63
IBM Systems Technical Events | ibm.com/training/events be reproduced in whole or in part without the prior written permission of IBM.
RSH is dead.
Use nimsh
RSH is DEAD LONG LIVE NIMSH!

• Starting with AIX 6.1 TL8 and AIX 7.1 TL2, the nimadm tool now supports the NIM service
handler (nimsh) protocol.
• Using this protocol allows AIX migrations to be performed without enabling rshd on the
NIM client.
• These articles describe how to use nimsh with nimadm to migrate to AIX version 7.1:
• http://www.ibm.com/developerworks/aix/library/au-nim_service_handler/
• http://www.ibmsystemsmag.com/aix/administrator/systemsmanagement/nimsh_nimadm/
• https://www.ibm.com/developerworks/community/blogs/cgaix/entry/rsh_is_dead_long_live
_nimsh?lang=en
• Based on my tests, using nimsh adds around 20 minutes to the overall migration time,
when compared with rsh. Your mileage may vary.
NIMSH+NIMADM SHELL+NIMADM
============ ============
Start: Fri Aug 30 14:37 EET 2013 Start: Sun Sep 1 20:01 EET 2013
Finish: Fri Aug 30 15:47 EET 2013 Finish: Sun Sep 1 20:53 EET 2013
1 hr 10 min 52 min

• Yes, RSH is still supported with nimadm today but long term support for it will
(eventually) disappear!
.
IBM Systems Technical Events | ibm.com/training/events
© Copyright IBM Corporation 2018. Technical University/Symposia materials may not
be reproduced in whole or in part without the prior written permission of IBM.
64
Other tips and tricks
NIMSH, SSL, LPM & mksysb
I used LPM to move a NIM client, now my NIM mksysb’s don’t work! L

• CPUid changes after LPM operation. Sure does! Working as designed.


• Everything is OK until you restart the nimsh daemon. Cached CPUid is replaced.
• Use SSL with NIM. The CPUid is not used (ignored). SSL cert’s instead.

https://www.ibm.com/developerworks/community/blogs/cgaix/entry/NIMSH_SSL_and_LPM?lang=en
https://www.ibm.com/developerworks/community/blogs/cgaix/entry/NIMSH_SSL_Hang?lang=en*

[root@nimmaster]/ # nimconfig –c
[root@nimmaster]/ # /usr/samples/nim/ssl/certview -I /tftpboot/server.pem

[root@AIXcg]/# nimclient –c
[root@nimmaster]/ # lsnim -l AIXcg
AIXcg:
class = machines
type = standalone
connect = nimsh (secure)

[root@AIXcg]/var/adm/ras# tail -f nimsh.log


Fri Apr 29 23:13:33 2016 passing OpenSSL setting of 1
Fri Apr 29 23:13:33 2016 set symbol table
Fri Apr 29 23:13:33 2016 cert filename discovered: /ssl_nimsh/certs/nimmast.0
Fri Apr 29 23:13:33 2016 ** OpenSSL FIPS mode enabled successfully
Fri Apr 29 23:13:33 2016 seed_prng
Fri Apr 29 23:13:33 2016 Loading certificates..
Fri Apr 29 23:13:33 2016 Loading private key file..
Fri Apr 29 23:13:33 2016 create BIO
Fri Apr 29 23:13:33 2016 - SSL Connection verified successfully –

*Note: APAR IV93152: NIM PUSH OPERATION TO CLIENT HANG ON NIMSH OVER SSL
https://www-304.ibm.com/support/entdocview.wss?uid=isg1IV93152
© Copyright IBM Corporation 2018. Technical University/Symposia materials may not 65
IBM Systems Technical Events | ibm.com/training/events be reproduced in whole or in part without the prior written permission of IBM.
Other tips and tricks
NIM and VIOS support
VIOS to NIM Master Mapping.
• Minimum NIM master levels for VIOS NIM clients.
• Your NIM master must always be at the highest level of AIX that it's required to support.
• https://www-304.ibm.com/webapp/set2/flrt/sas?page=viostable

© Copyright IBM Corporation 2018. Technical University/Symposia materials may not 66


IBM Systems Technical Events | ibm.com/training/events be reproduced in whole or in part without the prior written permission of IBM.
Known issues related to nimadm and/or AIX migration
• If you're planning to migrate from AIX 7.1 TL5 to AIX 7.2, using nimadm, please review the following
APARs first. An ifix is available for both APARs (for AIX 7.1 TL5 SP1). You can request the ifix(es) from
IBM support. Workaround: Use rsh instead of nimsh.

IJ01500: NIMADM ISSUE WITH RPM COMMAND


http://www-01.ibm.com/support/docview.wss?uid=isg1IJ01500

IJ01499: NIMADM FAILS DUE TO SOME COMMANDS (AWK AND GREP) NOT FOUND
http://www-01.ibm.com/support/docview.wss?uid=isg1IJ01499

• nimadm with cachevg fails in updating RPMs. Request an ifix.

IJ02912: NIMADM WITH CACHEVG MAY FAIL IN UPDATING RPM WITH CHFS ERROR
https://www-01.ibm.com/support/entdocview.wss?uid=isg1IJ02912

Error description
nimadm with cachevg may fail in updating rpm with chfs error.

install_all_updates: Updating rpm packages..


Validating RPM package selections ...
chfs: No record matching '/dev/fslv04' was found in /etc/filesystems.

Local workaround
Use nimadm without cache VG

© Copyright IBM Corporation 2018. Technical University/Symposia materials may not 67


IBM Systems Technical Events | ibm.com/training/events be reproduced in whole or in part without the prior written permission of IBM.
Known issues related to nimadm and/or AIX migration
• Problem Description: AIX 7.1 TL4 SP3 to AIX 7.2 TL1 SP3 migration. The bos.cifs_fs.rte is down
level. Problem Resolution: Fileset (bos.cifs_fs) is available in the AIX 7.2 Expansion Pack. Download
ISO from ESS.

https://www.ibm.com/support/knowledgecenter/en/ssw_aix_72/com.ibm.aix.rnbase720/rnbase720.htm

"CIFS Client fileset


The bos.cifs_fs software package has moved to the AIX Version 7.2 Expansion Pack media. The CIFS
Client is provided as-is (without support)."

To install the base fileset you need to set the installp 'force overwrite' option ( -F), so run:

# installp -d . -acFXY bos.cifs_fs.7.2.0.0.I

Once the base fileset is installed, run 'smitty update_all" again to bring the fileset up to the latest level
for AIX 7.2.

© Copyright IBM Corporation 2018. Technical University/Symposia materials may not 68


IBM Systems Technical Events | ibm.com/training/events be reproduced in whole or in part without the prior written permission of IBM.
AIX blogs and other online resources

• Using NIM Alternate Disk Migration (NIMADM)


• http://www-01.ibm.com/support/docview.wss?uid=isg3T1012571

• AIXchange (Rob McNelly).


• http://ibmsystemsmag.blogs.com/aixchange/
• http://ibmsystemsmag.com/blogs/aixchange/archive/using-mksysb-for-nim-backups/

• Chris’s AIX blog.


• https://www.ibm.com/developerworks/mydeveloperworks/blogs/cgaix/?lang=en
• https://gibsonnet.net/AIX/

• AIX articles on IBM developerWorks.


• http://www.ibm.com/developerworks/views/aix/libraryview.jsp?sort_by=&show_abstract=tr
ue&show_all=&search_flag=&contentarea_by=AIX+and+UNIX&search_by=chris+gibson
&topic_by=-1&industry_by=-1&type_by=All+Types&ibm-search=Search

• AIX articles on IBM Systems Magazine


• http://ibmsystemsmag.com/authors/chris-gibson/

• Follow AIX resources on Twitter: @cgibbo, @Kristijan, @robmcnelly, @chet_metha,


@IBMAIXdoc, @power_gaz, @mr_nmon
© Copyright IBM Corporation 2018. Technical University/Symposia materials may not 69
IBM Systems Technical Events | ibm.com/training/events be reproduced in whole or in part without the prior written permission of IBM.
IBM Systems Lab Services & Training - Power Systems

AIX 7.2 Migration with NIMADM


Overview
The NIM Alternate Disk Migration tool is a significant feature of the AIX
Duration:
Enterprise offering that allows customers to migrate their AIX systems to a
* 1 – 2 day engagement. Customers and Business Partners preparing to implement NIMADM on AIX7.2 and
newer release with minimal downtime.
achieve minimal application downtime when migrating. POC (Proof of Concept).

Conventional AIX migration techniques often require significant downtime; Delivery:


from anywhere between 30-45 minutes to several hours. This can be a
serious concern in environments with tight outage windows and lack of high This is an onsite offering consisting of standup lectures. Presentation handouts are provided in PowerPoint
availability or clustering. The requirement for lengthy application outages often format.
led to delays in migrating to a new AIX release, leaving customers unable to If the scope of this offering is to implement NIMADM into an existing customer or POC environment, IBM will
take advantage of the latest features and enhancements. conduct a design and requirements review to insure successful migration to AIX 7.2 of a partition..

Topics:
This offering covers the concepts, implementation and configuration of the
NIM alternate disk technology.
* Overview of traditional migration methodologies
This offering provides a customer or business partner knowledge and hands- * Introduction of NIMADM concepts, design and comparison to traditional OS update technologies
* Introduction of NIMADM requirements and Configuration options
on experience to make an intelligent decision when implementing NIMAD over
* Configure AIX NIM master for NIMADM and examining resource requirements
traditional migration strategies. This offering is a quick, cost effective method
to integrate this technology into an existing Power environment, training IT
Staff and obtaining the best availability possible from IBM Power architecture.

Who benefits from this workshop?

* Customers and Business Partners considering NIMADM over traditional


migration strategies.

* Customers looking to quickly implement NIMADM into existing Power7 and


Power8 environments and train their IT staff

* Business partners requiring the knowledge and skills necessary to assist


customers migrating to AIX 7.2 with NIMADM.

* Business partners and customers performing POC engagements to evaluate


AIX 7.2 NIMADM migration and integrate into existing customer procedures and
environments.

Terms and Conditions: Actual Tasks, Deliverables, Service Estimates, and travel requirements vary with each client’s environment. When we have reached a final agreement on the scope of your initiative and our level of assistance, a formal
document describing our proposed work effort, costs, etc, will be presented for your approval and signature.

IBM Systems Lab Services & Training - Power Systems Jacell Bangit – Opportunity Manager jbangit@au1.ibm.com +61409 271 375
Services for AIX, and Linux on Power

70
Copyright IBM Corporation 2017
Migrating to AIX 7.2 with minimal downtime

THANK YOU!
ANY QUESTIONS?
cgibson@au1.ibm.com

?
IBM Systems Technical Events | ibm.com/training/events
© Copyright IBM Corporation 2018. Technical University/Symposia materials may not
be reproduced in whole or in part without the prior written permission of IBM.
71

You might also like