Professional Documents
Culture Documents
IT Operation Excellence
InfraOptimization
Presenter Profile
Name Md Mozammel Hoque
Designation CEO & BPO Consultant at InfraOptimization
Career at a glance 17 years+ as Executive leader, Consultant, Researcher, Enterprise Architect, Program Manager, and Software Engineer
in Telco, Banking, Manufacturing and Retail Industries. He focuses on customer experience, Business strategy and
Process Optimization, innovation, leadership, and improved operational efficiency. He supported more than 70
Clients (on site and off-shore) with hundreds of projects Implementer and Designer on Oracle ERP, Telco Billing,
DWH, Campaign Management, Fraud and Revenue Management, Security and Audit, covering North America,
Europe, West Africa and Asia. He worked as (Assistant) Researcher on the field of Socio-Anthropology, and had
teaching experience.
He has served as CEO and BPO Consultant at InfraOptimization, as Management & Strategy Consultant at E-Tec Ltd.,
as CTO at NTS, as Sr. Technical Consultant at Huawei (Base Zone: West Africa), as Oracle Trainer at IBCS-PRIMAX and
as APPS DBA at DatabaseIntelligence (Russia) on behalf of DataIntensity (USA).
Publication A number of Technical and Non-Technical articles were published in 3MS/HUAWEI Case Center.
Introduced 7 RIGHTS concept for better business output, which was disclosed (and organized by) at the Faculty of
Business Administration, University of Chittagong, Bangladesh on May 25, 2016.
Public Speaker He has been the speaker on Optimization: Business and IT hosted by the University of Chittagong, Bangladesh and
IBCS-Primax (Dhaka, Bangladesh).
Professional Certification Oracle Certified Expert (OCE) - Oracle RAC 11g Release 2 and Performance Tuning, Oracle Certified Specialist (OCS) -
Oracle Database 11g Data Warehouse, Oracle Certified Professional (OCP) - R12, 11g, 10g and 9i.ITIL V3 Foundation
Certified from EXIN. Passed Total 31 exams with all distinct result i.e. Database, Operating System, Network, Security,
OCS/CBS, Storage, ITIL etc. in HUAWEI exam system.
Reward Honorable Performance Reward for the year of 2013, Best performance award for the year of 2012 for Etisalat Nigeria
project from Huawei, and Two times best performance award (2010 and 2011) from Zain (now Airtel) Telco in Nigeria
on behalf of International Turnkey System.
Education MSC and BSC in Applied Informatics in Economics from Rostov State University of Civil Engineering, Russia in 2006
MA in 1997 and BA in 1996 in History from the University of Dhaka, Bangladesh
- What to Know | Do
Process:
Key Learning Principle
- Why to know | Do
InfraOptimization
Content of Training
Forms of Training
Lecture: Concept
Lab: Practical
Case Study: Live & Past
InfraOptimization
Applications Technology Release 12.2 Value Proposition
Oracle E-Business Suite Applications Technology provides the technology infrastructure to
develop, manage, and integrate Oracle E-Business Suite applications.
The Applications Technology team incorporates and adapts products from Oracle Database
and Oracle Fusion Middleware to suit the unique requirements of Oracle E-Business Suite.
The breadth and depth of the technology adoption is extensive, including:
InfraOptimization
Applications Technology Release 12.2 Value Proposition Cont 1
The result is that Oracle E-Business Suite offers unparalleled productivity, scalability,
performance, and availability - thereby reducing the total cost of ownership (TCO) for
enterprises of all sizes.
Oracle E-Business Suite Applications Technology Release 12.2 is focused on the following
strategic areas:
• Technology Update
• Usability
• Integration
• Security
• Manageability
Source:
ORACLE E-BUSINESS SUITE RELEASE VALUE PROPOSITION
Release 12.2 Applications Technology
Prepared by E-Business Suite Product Management and Strategy
Last Updated: September 18, 2013
Version: 2.0
InfraOptimization
Reference Doc IDs on EBS R12.2.x
Oracle E-Business Suite Release 12.2 Information Center (Doc ID 1581299.1)
Release Schedule of Current Database Releases (Doc ID 742060.1)
https://blogs.oracle.com/stevenchan/resource/certifications.html
Oracle E-Business Suite Release 12.2 Technology Stack Documentation Roadmap (Doc ID 1934915.1)
Oracle E-Business Suite Release Notes, Release 12.2 (Doc ID 1320300.1)
Oracle E-Business Suite Product Specific Release Notes, Release 12.2.3 (Doc ID 1605928.1)
Oracle E-Business Suite Release 12.2.3 Readme (Doc ID 1586214.1)
Oracle E-Business Suite Technology Stack and Applications DBA Release Notes for Release 12.2.3 (Doc
ID 1606170.1)
Oracle E-Business Suite Product Specific Release Notes, Release 12.2.4 (Doc ID 1903217.1)
Oracle E-Business Suite Release 12.2.4 Readme (Doc ID 1617458.1)
Oracle E-Business Suite Product Specific Release Notes, Release 12.2.5 (Doc ID 2049015.1)
Oracle E-Business Suite Release 12.2.5 Readme (Doc ID 1983050.1)
Oracle E-Business Suite Product Specific Release Notes, Release 12.2.6 (Doc ID 2114173.1)
Oracle E-Business Suite Release 12.2.6 Readme (Doc ID 2114016.1)
Oracle E-Business Suite Release 12.2: Technical Planning, Getting Started, and Go-Live Checklist (Doc ID
1585857.1)
InfraOptimization
Concepts and Terminology:
Tier, Node, Service/Server
A Tier is a logical grouping of services, potentially spread across more than one
physical machine. The three-tier architecture that comprises an Oracle E-Business
Suite installation is made up of the database tier, which supports and manages the
Oracle database; the application tier, which supports and manages the various
Oracle E-Business Suite components, and is sometimes known as the middle tier;
and the desktop tier, which provides the user interface via an add-on component to
a standard web browser.
Desktop or Client Tier - provides the user interface using a web browser to access the applications.
Application Tier - The application tier, sometimes referred to as the middle tier, hosts the various services and
servers that implement and execute the business logic, and mediates communications between the desktop tier
and the database tier.
Database Tier - supports the Oracle RDBMS and is the main data storage area. The database server does not
communicate directly with desktop clients
EBS R12.2 Application Tier Structure
Source: Oracle E-Business Suite Concepts Release 12.2 Part Number E22949-12
InfraOptimization
EBS R12.2 Application Tier Services at-a-glance
oacore
oafm
Web Application Services admanagedsrvctl.sh
forms
forms-c4ws
Most notably, the Web Administration service group contains WebLogic Administration server, and - unlike other
service groups - can be enabled only on one of the Application tier nodes. In other words, it is not supported to
enable WebLogic Administration server on any other Application tier node except the node on which it was
enabled during Rapid Install.
InfraOptimization
R12.2 Technical Configuration
Configuration Activity In Release R12.2 Configuration Management Tools
InfraOptimization
EBS Release 12.2 Technology Stack
FM 11g
End User
InfraOptimization
EBS Release 12.2 Architecture
From Weblogic Perspective: WHAT TO KNOW
InfraOptimization
EBS Release 12.2 Architecture
From Weblogic Perspective: WHAT TO KNOW ON SCALING
InfraOptimization
EBS R12.2 Deployment Planning Consideration
1. Performance
2. Scalability – Vertically | Horizontally
3. Maximum Availability
4. Ease of Management & Maintenance
InfraOptimization
EBS R12.2 Deployment Architecture
Load balancing: DNS Load balancing: DNS| HTTP (using load balancer)
InfraOptimization
EBS R12.2 Deployment Design Decisions
InfraOptimization
EBS R12.2 Deployment Design Decisions
MOS Doc ID 1375686.1
InfraOptimization
EBS R12.2 Deployment Design Decisions
InfraOptimization
EBS Release 12.2 Technology Stack Certification
Source:
ORACLE E-BUSINESS SUITE RELEASE CONTENT DOCUMENT
Applications Technology Releases 12.1 and 12.2 (inclusive of 12.0.2 – 12.0.7)
Prepared by ATG Documentation
Last Updated: March 6, 2015
Version: 3.0
InfraOptimization
InfraOptimization
Pathway to Sizing for EBS R12.2
Considering Factor for Sizing for EBS R12.2
Number of concurrent E-Business Suite users, i.e. Oracle Application Testing Suite
Sizing information systems: SPC - Storage Performance Council & TPC - Transaction
Processing Performance Council
InfraOptimization
EBS R12.2 Architecture & File System Structure
Source: Sharing The Application Tier File System in Oracle E-Business Suite Release 12.2 (Doc ID 1375769.1)
General Three Tier Architecture Application/Database Tier Components
InfraOptimization
EBS R12.2 File System Structure
Source: https://docs.oracle.com/cd/E26401_01/doc.122/e22949/T120505T120509.htm (Part Number E22949-12)
InfraOptimization
EBS R12.2 File System Structure Cont 1
• 10.1.2 OH holds the AS
components for Forms
and Reports
• Fusion Middleware
Home contains
WebLogic Server, HTTP
configuration files
• INST_TOP contains only
maintenance scripts
• Fusion middleware home contains the following directories
– EBS_Domain: E-Business Suite weblogic domain, contains all domain-specific configuration and log files.
– wlserver_10.3: Binaries and libraries required to support WebLogic Server
– oracle_common: Java Required Files (JRFs) used by EBS.
– webtier: HTTP configuration and log files used by EBS.
– Oracle_EBS-app1: Contains configuration files related to oacore, forms, oafm, and forms-c4ws
• FS_NE contains files needed by both the file systems
– Patch staging area, ADOP log files & ADOP utility
• Not all log files are not stored under $INST_TOP
– HTTP log files are located under the native instance home ($FMW_HOME/webtier)
– Oracle WebLogic Server log files are located under the domain home ($EBS_DOMAIN_HOME)
– Concurrent processing log files are located on the none-editioned file system (fs_ne)
InfraOptimization
Deciding RAID Levels for EBS R12.2 File System
Source: https://en.wikipedia.org/wiki/Standard_RAID_levels
InfraOptimization
Oracle Applications Technology Layer
The Oracle Applications technology layer is an integrated collection of components whose
functionality is applicable to all Oracle Applications modules, i.e. CRM, HRMS, Financials,
Manufacturing. Components in the Oracle Applications technology layer include:
InfraOptimization
Oracle Applications Database Objects
• Data objects - Used for storing and accessing business data. These objects include tables,
indexes, sequences, and index-organized tables.
• Code objects - Used to process the data. These objects include triggers, PL/SQL packages,
Java stored procedures, synonyms, and views.
• Hybrid objects - Used to both store and process data. Materialized views are an example of
hybrid objects.
InfraOptimization
Oracle Applications Schemas
(Global Schema)
Product Schema and its own objects a better name for APPLSYS would or could
be FND
Code Objects
InfraOptimization
Purpose of 2 applsyspub entries in context file
InfraOptimization
EBS R12.2 Login Process Doc ID: 1569732.1, 2072420.1,
Reference Doc ID 1984710.1 1316142.1
Oracle E-Business Suite Concepts Release 12.2 Part Number E22949-12
http --> HS/mod_wl_ohs
Oracle Application Framework Architecture --> http --> WLS
s_wls_admin_
user
Session Tables
ICX_SESSIONS
FND_LOGINS
EBS R12.2 Login Process
Reference Doc ID 1984710.1
WLS determines the J2EE application that should deal with the request, which is
called "oacore".
This J2EE application needs to be deployed and available for processing requests
in order for the request to succeed. The J2EE application needs to access a
database and does this via a datasource which is configured within WLS. s_wls_admin_
user
All code must access the data model via the APPS synonym!
Session Tables
ICX_SESSIONS
ICX_PARAMETERS
FND_LOGINS
Differences between Forms Server and Forms Servlet
Source: Troubleshooting Forms Applet issues in E-Business Suite (Doc ID 1300317.1)
Reference: Using Forms Socket Mode with Oracle E-Business Suite Release 12 (Doc ID 384241.1)
Although Forms Server (or SOCKET mode) and Servlet implementation mainly affects the Application tier there is some
important differences on how communication is done between the Forms applet and run-time process making it useful to
provide some background information on this topic.
EBS R12.2.x Password management
Steps to change the APPS, APPLSYS, and APPS_NE password using FNDCPASS or AFPASSWD for EBS 12.2 (Doc ID
1674462.1)
source EBSapps.env run ; env |grep -i base, grep BASE $CONTEXT_FILE
• FNDCPASS and AFPASSWD utilities can be used for setting Oracle E-Business Suite schema passwords
• It is recommended that the FND_USER and FND_ORACLE_USERID tables are backed up before system passwords (APPLSYS, APPS,
APPS_NE) are changed
• AFPASSWD is an enhanced version of FNDCPASS, only prompts for passwords required for the current operation.
AFPASSWD [-c <APPSUSER>[@<TWO_TASK>]] -f <FNDUSER> (Application Users)
AFPASSWD [-c <APPSUSER>[@<TWO_TASK>]] -o <ORAUSER> (Apps Product users)
AFPASSWD [-c <APPSUSER>[@<TWO_TASK>]] –a (All Apps Product users)
AFPASSWD [-c <APPSUSER>[@<TWO_TASK>]] -l <ORAUSER> <TRUE|FALSE> (Locks Single User)
AFPASSWD [-c <APPSUSER>[@<TWO_TASK>]] -L <TRUE|FALSE> (Locks non-essential users)
AFPASSWD [-c <APPSUSER>[@<TWO_TASK>]] -s <APPLSYS> (Changes APPLSYS)
AFPASSWD [-c <APPSUSER>[@<TWO_TASK>]] -m <HASH-MODE> [MIGRATE] (Migrated to non-recoverable Hashing
scheme passwords)
AFPASSWD -c apps@PROD -s APPLSYS
AFPASSWD -f SYSADMIN Enter the ORACLE password of Application Object Library
Enter the ORACLE password of Application Object Library ‘APPSUSER’: ‘APPSUSER’:
Connected successfully to APPS. Connected successfully to APPS.
Working… Enter the password for your ‘SYSTEM’ ORACLE schema:
Enter new password for user [SYSADMIN]: Connected successfully to SYSTEM.
Verify new password for user [SYSADMIN]: Working…
Password is changed successfully for user SYSADMIN. Enter new password for user:
Password is changed successfully for user SYSADMIN. Verify new password for user:
AFPASSWD completed successfully. Working…
AFPASSWD completed successfully.
Note: For Changing APPLSYS/APPS/APPS_NE Password We Need To Shutdown Application Tier And Once We Finish
Changing The Password We Need To Run Autoconfig On All Tiers
InfraOptimization
EBS R12.2.x APPS Password change
• Shut down the application tier services
• Change the APPLSYS password using FNDCPASS or AFPASSWD utility.
• Start AdminServer using the adadminsrvctl.sh script from your RUN filesystem
• Update the “apps” password in WLS Datasource as follows:
– Log in to WLS Administration Console.
– Click Lock & Edit in Change Center.
– In the Domain Structure tree, expand Services, then select Data Sources.
– On the “Summary of JDBC Data Sources” page, select EBSDataSource.
– On the “Settings for EBSDataSource” page, select the Connection Pool tab.
– Enter the new password in the “Password” field.
– Enter the new password in the “Confirm Password” field.
– Click Save & Click Activate Changes in Change Center.
• Execute Autoconfig to update all application tier files
• Start all the application tier services using the adstrtal.sh script.
Doc ID 1377400.1 - Install Sanity Check: How To Deploy A Basic Java Web Application To Confirm Your WebLogic Domain Is Functional
Doc ID 1940996.1 - Oracle E-Business Suite 12.2 Data Source Connection Pool Diagnostics
InfraOptimization
OAM Generic Collection Service concurrent manager
Source:
How To Determine The Current Value and Set The Retention Time To Keep the Runtime Instance Data Of Table
FND_OAM_FORMS_RTI In The Database? (Doc ID 1600075.1) & What Are .RTI And .FLI Files? (Doc ID 470850.1)
How To Find The IP Address Of The Client Machine From Where A Particular Forms User Is Connected ? (Doc ID
879092.1)
env|grep FORMS_RTI_DIR
Notes:
Data in table FND_OAM_FORMS_RTI is populated by the OAM Generic Collection Service concurrent manager and the
information is cleaned up and repopulated by the same program. The OAM Generic Collection Service takes these parameters:
NODE: the name of the node on which the service runs.
LOADINTERVAL: the load interval for periodic runtime instance information uploading.
ORACLE_HOME: the ORACLE_HOME in which the Forms Listener runs.
RTI_KEEP_DAYS: the number of days to keep the runtime instance data in the database.
FRD_KEEP_DAYS: the number of days to keep Forms Runtime Log files.
InfraOptimization
Oracle E-Business Suite SYS.DUAL PUBLIC Privileges Security Issue (CVE-2015-0393)
Source:
Use of Common Vulnerability Scoring System version 2.0 (CVSS 2.0) by Oracle (Doc ID 394487.1)
Oracle Critical Patch Update January 2015 Documentation Map (Doc ID 1935468.1)
SELECT 'revoke ' || privilege || ' ON sys.dual FROM public;' "SQL" FROM sys.dba_tab_privs WHERE
owner = 'SYS' AND table_name = 'DUAL' AND grantee = 'PUBLIC' AND privilege != 'SELECT';
Note:
For R12 only, the CPU patch has manual instructions to run the adgrants.sql script. This script will create
the AD_ZD_SYS package in the SYS schema. AD_ZD_SYS has a single procedure FIX_SYSUSER, which
revokes unnecessary grants on SYS.DUAL.
InfraOptimization
Monitoring & Auditing User Activity
E-Business Applications Performance Basic Skills Cookbook (Doc ID 1516355.1)
Concurrent Request Data Gathering (Doc ID 740770.1)
APXINWKB APP-10771 Could Not Reserve Records Due to Database Record Lock (Doc ID 140969.1)
How To Kill Inactive Sessions Based on the Information in the Monitor User Form (Doc ID 103516.1)
Oracle Critical Patch Update January 2015 Documentation Map (Doc ID 1935468.1)
Use of Common Vulnerability Scoring System version 2.0 (CVSS 2.0) by Oracle (Doc ID 394487.1)
Oracle Database Security Assessment Tool (DBSAT) (Doc ID 2138254.1)
How to find or see the requests ran in a module or a responsibility or by a user or for a specific period
(Doc ID 2098951.1)
Monitor Users: System Administrator Responsibility Security: Users > Monitor CTL + F11
Enable User Tracking – Setting the Sign-On: Audit Level profile option:
InfraOptimization
Technology Stack Registration
Global and Local Inventory explained (Doc ID 360079.1)
FAQs on Central Inventory and Oracle Home Inventory (Local Inventory) in Oracle RDBMS (Doc ID 564192.1)
How To Create, Update or Rebuild The Central Inventory For Oracle Applications E-Business Suite Release 12 ? (Doc ID
742477.1)
oraInst.loc oraInst.loc
inventory_loc=/etc/oraInventory Under /etc on Linux and AIX
Under /var/opt/oracle on Solaris and HP-UX
InfraOptimization
Oracle Universal Installer Inventory Essentials for Apps Sysadmins
Source: https://blogs.oracle.com/stevenChan/entry/ebs_oui_inventory_primer
InfraOptimization
Oracle Universal Installer Inventory Essentials for Apps Sysadmins
Source: https://blogs.oracle.com/stevenChan/entry/ebs_oui_inventory_primer
Oracle Universal Installer Inventory Essentials for Apps Sysadmins
Source: https://blogs.oracle.com/stevenChan/entry/ebs_oui_inventory_primer
Cloning
Cloning is the process used to create a copy of an existing E-Business Suite System
InfraOptimization
Pre-Requisite Before Cloning
Reference Doc ID: 1383621.1
Before cloning to a new system, you must first prepare the Source System by following the
below steps:
InfraOptimization
Pre-Requisite Before Cloning
Cleaning Environments
InfraOptimization
Cloning
Techniques
InfraOptimization
Cloning Techniques
InfraOptimization
Cloning Methods
Off-Line
On-Line/ Hot Cloning/AMP Cloning
Note:
Apps Tier can be cloned on-line at any point in time.
DB Tier, i.e. Database, can be cloned on-line by RMAN only.
InfraOptimization
Cloning Techniques
Rapid Clone
Reference Doc ID
• Cloning Oracle Applications Release 12 with Rapid Clone (Doc ID 406982.1)
• Cloning Oracle E-Business Suite Release 12.2 with Rapid Clone(Doc ID 1383621.1)
• Setup Guide: Part No. E22953-10 (Chapter 17 Cloning Oracle E-Business Suite with Rapid Clone)
• How To Collect Log Files from Cloning in 12.2 (Doc ID 1961471.1)
• E-Business Suite Clone Log Parser Utility (Rapid Clone) (Doc ID 1447553.1)
• Cloning Oracle Applications Release 11i with Rapid Clone (Doc ID 230672.1)
• Troubleshooting RapidClone issues with Oracle Applications R12.0 & R12.1 (Doc ID 603104.1)
• http://docs.oracle.com/cd/E11857_01/em.111/e18709/T508706T512065.htm
InfraOptimization
Cloning Techniques
Rapid Clone at EBS Application Tier
/appsprod/fs1 /appstest/fs1
Up To R12.2.x-4 PATCH PATCH
Critic on this Clone Architecture: Repetitive Task
Source Target
1. Clone (Source RUN FS): Source to Target
2. Clone (Target RUN FS): Target RUN FS to Target PATCH FS.
From R12.2.5-6
1. Clone (Source RUN FS): Source to Target
InfraOptimization
Cloning Phases
Prepare Source System by applmgr & oracle user, NOT root
$ cd <RDBMS ORACLE_HOME>/appsutil/scripts/<CONTEXT_NAME>
$ perl adpreclone.pl dbTier
$ cd $INST_TOP/admin/scripts
$ perl adpreclone.pl appsTier
InfraOptimization
Cloning Phases
Copy Bits by ‘application’ OS user, i.e. NOT root user
Note:
Comparing the size of the file systems between Target and Source
InfraOptimization
Cloning Phases
Configure Target System
## As Oracle User
cd $ORACLE_HOME/appsutil/clone/bin
perl adcfgclone.pl dbTier /* perl adcfgclone.pl <component> [context] */
## As applmgr user
…/EBSapps/comn/clone/bin
perl adcfgclone.pl appsTier
WHERE:
<component> on Apps Node : { appsTier | atTechStack | dev10gHome | fmwHome | wlsConfig | ohsConfig | appltop }
<component> on DB Node : { dbTier | dbTechStack | database | dbconfig | addracnode }
<context> : Target Context File
Cloning Techniques
Configure Target System
Configclone dbTier
Configclone appsTier
InfraOptimization
Cloning Techniques
Other Configuration Commands
InfraOptimization
Cloning Techniques
Other Configuration Commands
InfraOptimization
Cloning Techniques
Custom Cloning: Source-to-Target Refreshing
Target System
InfraOptimization
Cloning Techniques
Maximum Availability Techniques
Target System
InfraOptimization
AutoConfig:
To Manage EBS System Configurations
You must run AutoConfig for your system to make any context file updates to your
system
Doing so generates the new configuration files for the associated technology stack
AutoConfig uses the parameters stored in the context file and system configuration
templates to create new process control scripts and update system profiles
• AutoConfig template files are used as the starting point for creating
site-specific configuration files. AutoConfig evaluates the context
variables in a template file, determines the actual values required, and
creates a configuration file with these values substituted. This process,
described in more detail later in this chapter, is called instantiation.
There is one template file for each configuration file. Template files are
located in the various <PROD>_TOP/admin/template directories on the AutoConfig Instantiation Mechanism
application tier, and in the
<RDBMS_ORACLE_HOME>/appsutil/template directory on the
database tier.
InfraOptimization
Reference Doc ID to Autocinfig
Note 387859.1 - Using AutoConfig to Manage System Configurations in Oracle E-Business Suite Release 12
InfraOptimization
Net Services Topology Data Model
The Net Services Topology Data Model stores the entire topological information about a single Oracle
Application instance. The data model stores information about each node in the Oracle Applications
instance which is then used to generate the Net Service configuration files (for example tnsnames.ora).
AutoConfig seeds the data model with relevant data.
The Net Services Topology Data Model stores the following information:
On the database tier: Hostname, Database SID, Database Name, Instance Name, TNS Descriptors.....
On the application tier: Hostname, FNDFS and FNDSM alias descriptors.........
The Net Services Topology Data Model is seeded every time you run AutoConfig on the respective tier.
Every time you run AutoConfig on the database tier, the relevant data is seeded/updated for the database
tier. Respectively, every time you run AutoConfig on the application tier, the relevant data is
seeded/updated for the application tier.
To deregister the current application tier from the Net Services Topology Data Model, invoke the
following command:
$ perl <AD_TOP>/bin/adgentns.pl appspass=<APPSpwd> contextfile=<CONTEXT> -removeserver
To deregister the current database tier from the Net Services Topology Data Model, invoke the following
command:
$ perl <RDBMS ORACLE_HOME>/appsutil/bin/adgentns.pl appspass=< APPSpwd>
contextfile=<CONTEXT> - removeserver
InfraOptimization
Context Files
XML based Context Files, uses by Autoconfig, is a Repository of all the installed Configuration Parameters with value: Application Context File and Database
Context File, i.e. hundreds of config and env files present in EBS.
Run the following script to determine if all context files have been registered in the database. The script should return a run and patch context
file for each application server and one for the database server.
select path,status,serial_number,version from fnd_oam_context_files where name not in ('TEMPLATE','METADATA') and (status is null or status !='VALID');
If the run file system data is missing from the database simply source the run file system environment and run autoconfig.
If the patch file system data (managed servers or context file) is missing from the database then run the following after sourcing the run file
system environment. This command is executed on the RUN file system to upload the PATCH context file:
By Autoconfig
Doc ID : 455492.1
Open the $APPL_TOP/admin/adovars.env file and
ensure that the environment variables below are
defined. Allows to sync multiple nodes
$ADJVAPRG –version
echo $CLASSPATH
$AFJVAPRG –version
echo $AF_CLASSPATH
Understanding AutoConfig
Reference Doc ID 387859.1
InfraOptimization
Using AutoConfig to Manage Oracle E-Business Suite Services
Service Group Service(s) Service Control Script
oacore
oafm
Web Application Services admanagedsrvctl.sh
forms
forms-c4ws
APPS: <INST_TOP>/appl/admin/<CONTEXT_NAME>.xml
Context File DB: <RDBMS_ORACLE_HOME>/appsutil/<CONTEXT_NAME>.xml
<PROD_TOP>/admin/template
Temp late File
<RDBMS ORACLE_HOME>/appsutil/template
Driver <AD_TOP>/admin/driver/
<FND_TOP>/admin/driver/
Another utility, the Performance Profiler will allow you to review and resolve performance related issues that
are experienced during the execution of autoconfig. Like the check config utility, you run the tool and analyze
the HTML output file.
InfraOptimization
AutoConfig Search Utility
https://blogs.oracle.com/stevenChan/entry/everything_you_ever_wanted_to_know_about_autoconfi
Using AutoConfig to Manage System Configurations in Oracle E-Business Suite Release 12 (Doc ID 387859.1)
The AutoConfig Search Utility (formally known as the TXK Context Variable Information Utility) scans your
AutoConfig infrastructure for a requested string and generates a detailed report about the relevant
AutoConfig variables and their usage. The report contains, among other things, the following information
about relevant AutoConfig variables:
1. Description
2. Current value
3. Default value
4. Relevant AutoConfig template files
perl $FND_TOP/bin/txkrun.pl -script=GenCtxInfRep -keyword="<keyword>"
InfraOptimization
Running AutoConfig in Parallel in EBS 12
Source: https://blogs.oracle.com/stevenChan/entry/power_tools_autoconfig_parallel_runs_in_ebs12
In an R12 E-Business Suite instance, AutoConfig can now be run simultaneously across multiple nodes.
This new feature significantly lowers maintenance downtime for multi-node installations.
The AutoConfig Build Context utility (adbldxml.pl) is needed like in case of cross-platform migrations and
database upgrades.
perl $ORACLE_HOME/appsutil/bin/adbldxml.pl
InfraOptimization
Oracle EBS R12.2:
Concurrent Processing Server
InfraOptimization
Concurrent Processing Server
InfraOptimization
Concurrent Manager:
Introduction
Batch Processing Sub-System
Programs are run as operating system background processes.
These programs may be written using variety of Oracle tools, programming languages for executables, or OS
scripts.
The Internal Concurrent Manager (ICM), which functions as the "boss" of all the other managers, i.e. controls all
other concurrent managers. It administers the startup and shutdown of managers as defined by their work shift,
monitors for process failure, and cleans up if a failure occurs.
Warning: You cannot alter the definition of the Internal Concurrent Manager.
The Conflict Resolution Manager (CRM) enforces rules designed to ensure that incompatible concurrent requests
do not run in the same conflict domain.
The Standard Manager as shipped with Oracle Applications will accept and run any concurrent requests. It has no
specialization. The Standard manager is active all the time; it works 365 days a year, 24 hours a day.
Warning:
You should not alter the definition of the Standard concurrent manager. If you do, and you have not defined
additional managers to accept your requests, some programs may not run. Use the Standard manager as a safety
net, a manager who is always available to run any request. Define additional managers to handle your installation
site's specific needs.
Monitoring Concurrent Manager
Are the
Managers up
and running ?
Concurrent Request – Lifecycle
High Level
Phase Activity
Disabled Program to run request is not enabled. Contact your system administrator.
Inactive On Hold Pending request is placed on hold, by selecting Hold in the Status field of the Request Details zone.
No Manager No manager is defined to run the request. Check with your system administrator.
Concurrent Requests, Programs and Processes
Concurrent Manager:
Tables
Table Content
FND_CONCURRENT_REQUESTS Details of user requests, including status,
start date, and completion date.
FND_CONCURRENT_PROGRAMS Details of concurrent programs, including
execution method, whether the program is
constrained, and whether it must be run
alone.
FND_CONCURRENT_PROCESSES Cross-references between concurrent
requests and queues, and a history of
concurrent manager processes.
FND_CONCURRENT_QUEUES Information about each of the concurrent
manager queues.
Concurrent Manager
FND_CONCURRENT_REQUESTS
Worker 1
Job Run SelectARunnable
EB
J Status
Running
Complete
Lock?
A
E Update
Select
Update
Update
Select AA
Set EUpdate
for
Set
Set
for Running
Complete
Update
Running
o
Running
B b Running
Worker 2
SelectARunnable
B
A Running
Complete
Job Run
Lock? Running
A X Update
Select for B
B
Update BUpdate
SelectSet
for Running
C
Worker 3
Lock?
Lock?
Job Run SelectC
Select Runnable
ARunnable
B
D
D
CA
X Select
Update CUpdate
SelectSet
for
for Update
Running
DB X Update
Select for
E
Worker 4
Job Run SelectNone
SelectC Runnable
ARunnable
B
D
Lock?
Lock?
C
A XX Select Update
Select for
for Update
Sleep
D
B X Update Set
Select
Select
Update
DDUpdate
for
for
Set Running
Update
Complete
Service Management
With Service Management, the Internal Concurrent Manager
(ICM) manages the various service processes across multiple
hosts. On each host, a Service Manager acts on behalf of the
ICM, allowing the ICM to monitor and control service processes
on that host. System administrators can then configure, monitor,
and control services though a management console which
communicates with the ICM.
InfraOptimization
Concurrent Manager:
Tuning Consideration
• 50%of performance tuning is in the business!
– Review long-running auto resubmitted jobs
– Review short/long-running jobs – selective parameters?
– Don’t purge jobs that the users then have to run again
• Points to note
– More managers/workers != more throughput (More workers = More
accesses)
– Avoid enabling an excessive number of managers/workers
• New systems: a start point is 1-2 target process per CPU
– Balance with resources required by online activity
– managers/workers
• Carefully consider reducing the CRM sleep time to 5 ~ 10 secs
• Set Concurrent: Force Local Output File Mode" to "Yes“
• Set CONC_PMON_METHOD=LOCK
InfraOptimization
Concurrent Manager:
Enhancing Performance – Best Practice
Tuning Concurrent Queue Parameters
• Sleep Seconds is the number of seconds your Concurrent manager waits between checking the list of
pending concurrent requests (concurrent requests waiting to be started).
Tips: High number, e.g. 120s avoids constant polls to check for new requests. Low number ,
e.g. 20s is good enough when number of submitted requests is very high.
Note: Concurrent Manager will not sleep at all if there are pending requests.
• Cache Size at least twice the number of target processes.
• Process Increase it if the pending/normal requests are high based on physical server
capacity
Backend queue concurrent manager details
• Purging & Defragmentation associated tables
• Gather Schema Statistics
Select rpad(user_concurrent_queue_name
,50,'-') "Queue Name",TARGET_NODE,
Workload Management
MAX_PROCESSES,running_processes,
nvl(to_char(SLEEP_SECONDS), 'Not Set')
• Specialized Concurrent managers
Sleep_Second,
nvl(to_char(cache_size),'Not
Output Post Processor – OPP
Set')Cache_size
from FND_CONCURRENT_QUEUES_VL fcq
• Java Heap Size
order by 1;
Transaction Managers
InfraOptimization
Concurrent Manager:
Tuning Generic Guideline
Throughput
To maximize throughput consider reducing the sleep time of the Conflict Resolution Manager (CRM). The default value is
60 seconds. You can consider setting to 5 or 10 seconds.
Avoid enabling an excessive number of standard or specialized managers. It can degrade the performance due polling on
queue tables (FND_CONCURRENT_REQUESTS...). You need to create specialized managers only if there is a real need.
Avoid enabling an excessive number of standard or specialized managers. It can degrade the performance due polling
on queue tables (FND_CONCURRENT_REQUESTS...). You need to create specialized managers only if there is a real need.
Truncate the reports.log file in log directory. Refer Note.844976.1 for more details
$APPLCSF/$APPLLOG/
$APPLCSF/$APPLOUT
InfraOptimization
Concurrent Manager:
Tuning Pending/Normal Requests
InfraOptimization
Concurrent Manager:
Tuning Pending/Standby Requests
InfraOptimization
Concurrent Manager:
Tuning Other Status Requests
Running / Normal
If the request is taking longer time, enable Database Trace and tkprof and
find out expensive DML.
Completed / Error
Check the request log file and take action accordingly
Inactive / No Manager
If “Standard Manager” has “Specialization Rules” defined which has rows with
“Include”. You must only define exclude. If you add any program with “Include”
options, other programs get excluded automatically.
InfraOptimization
Concurrent Manager:
Tuning Inactive/No Manager Requests
InfraOptimization
Troubleshooting Concurrent Manager:
Enable Tracing
Note:- It is not suggested as this generates trace for all the request submitted by all end- users.
Hence this could generate lots of trace files which is an additional overhead and degrade
performance.
InfraOptimization
Concurrent Manager:
Tuning Options
Cache size
• Cache size = The number of pending requests being fetched for execution from the requests queue
• What cache size to choose?
• Small cache sizes are better for managers executing long running requests
• Small cache sizes are better for managers with lots of processes
• Larger cache size is good for managers with few processes executing quick (or both quick
and slow) requests
• Larger cache size is good if you have a lot of new requests incoming all the time
• If at least one request was fetched, the queue will be checked again right after all the cached requests
have been processed
• This reduces the significance of the cache size
• Guideline for tuning:
• Increase the cache size (number of requests cached) to at least twice the number of target
processes. For example, if a manager's work shift has 1 target process and a cache value of
3, it will read three requests, and try to run those three requests before reading any new
requests.
Tip: Enter a value of 1 when defining a manager that runs long, time-consuming jobs, and a
value of 3 or 4 for managers that run small, quick jobs.
This is only guidance and a balance needs to struck in tuning the cache, so with fast jobs you
need to cache to get enough work for a few minutes. With slow jobs, a small queue helps
should you need to reprioritize requests.
InfraOptimization
Concurrent
Requests
At least 1 job
InfraOptimization
RRA - Report Review Agent (FNDFS )
Online viewing of report log and output files
Viewing of reports one page at a time
Storage of report output on the concurrent processing node
The log or output file that results from a concurrent request is sent to
a utility known as the Report Review Agent, which passes a file
containing the entire report to the Forms services, which passes the
report back to the user’s browser one page at a time.
InfraOptimization
Output Post Processor - OPP
Concurrent Processing uses the Output Post Processor (OPP) to enforce Post Processing actions for concurrent requests.
Tip :
It is suggested to increase the number of "threads" as far as you can until you start hitting the memory limit. If you still need more
throughput, then you can increase "Processes". If you run lots of "Huge Reports" and running into OutOfMemoryError then you can
increase the "Processes".
The limiting factor with increasing "threads" is memory, if you set the max heap size of a JVM to 2GB, then all "threads" will share this
memory.
Concurrent -> Manager -> Define > "Output Post Processor"
oracle.apps.fnd.cp.opp.OPPServiceThread:2:0:max_threads=5
Increase Number of
OPP processes
OR
Increase the
number of
Threads
Output Post Processor - OPP
Heap Size
To prevent the error "java.lang.OutOfMemoryError: Java heap space“ check the current value :
Increase the maximum Java heap size for the OPP to 1024MB ( Max 2GB):
Error
The Output Post-processor is running but has not picked up this request. No further
attempts will be made to post-process this request, and the request will be marked with Warning status.
Action
Setting the Profile Option ‘Concurrent: OPP Response Timeout’ to a higher value may be necessary.
If the value of the Profile Option ‘Concurrent:OPP Response Timeout’ is 120 then increase it to 240.
(Value * 2).
InfraOptimization
Output Post Processor - OPP
Error
The Concurrent Manager has timed out waiting for the Output Post Processor to finish this
Request.
Check that there are enough Output Post Processor Service Processes running. More information
may be found in the Service Process logfile.
Action
Check the Profile Option "Concurrent:OPP Process Timeout".
Increase Increase the setting to a higher value.
For example: Set the value to 10800 sec (3 hours).
After this change, the Concurrent Request will wait maximum 3 hours, for the Output Post
Processor to finish the Request.
InfraOptimization
Oracle EBS R12.2
Data Model
InfraOptimization
EBS Data Model
Reference Doc
Oracle® E-Business Suite Concepts Release 12.2 Part No. E22949-12 September 2016
Oracle Applications Tablespace Model FAQs (Doc ID 269293.1)
Oracle E-Business Suite Release 12.2 Information Center - Implement (Doc ID 1583153.1)
https://support.oracle.com/epmos/main/downloadattachmentprocessor?parent=DOCUMEN
T&sourceId=1327399.1&attachid=1327399.1:EBS_SEED_DATA_VID&width=100%&height=10
0%
InfraOptimization
Database Features
Schema
product code objects (Packages, Procedures, Functions, Triggers, Views, Materialized views,
Java classes, Queues) are stored in the APPS schema, whereas product data objects (Tables,
Sequences, Indexes, Constraints, Queues) are stored in the relevant base product schemas, i.e.
GL, INV, AP etc.
Relationship Between APPS Schema and Base Product Schemas
Some views access packages or functions, where the value returned by the package or function may depend on the
environment having been set up properly. The environment is initialized automatically when accessing Oracle E-Business Suite
through the Sign-On screen, or when using concurrent processing with Oracle Reports or SQL scripts.
Consequently, if you connect directly to a schema, the rows returned by the view may be different from those returned when
you are running in an Oracle E-Business Suite environment. For example, a view may reference a profile option: when accessed
from SQL*Plus, the site value of the profile option will be used, rather than the setting for a particular Oracle E-Business Suite
user.
Space Management:
Oracle Applications Tablespace Model (OATM)
Oracle E-Business Suite Release 12.2 requires an Oracle database block size of 8K. No other
block size may be used.
Oracle E-Business Suite Release 12.2 utilizes as standard a modern infrastructure for tablespace management,
the Oracle Applications Tablespace Model (OATM). The Oracle Applications Tablespace Model is similar to the
traditional model in retaining the system, undo, and temporary tablespaces. The key difference is that Oracle E-
Business Suite products in an OATM environment share a much smaller number of tablespaces, rather than
having their own dedicated tablespaces.
Oracle E-Business Suite schema objects are allocated to the shared tablespaces based on two main factors: the
type of data they contain, and I/O characteristics such as size, life span, access methods, and locking granularity.
For example, tables that contain seed data are allocated to a different tablespace from the tables that
contain transactional data. In addition, while most indexes are held in the same tablespace as the base table,
indexes on transaction tables are held in a single tablespace dedicated to such indexes.
The Oracle Applications Tablespace Model provides a variety of benefits, which are summarized in the list below:
Simplifies maintenance and recovery by using far fewer tablespaces than the older model.
Uses locally managed tablespaces, whose choice of extent management types (autoallocate or uniform) allows
more precise control over unused space and hence can help reduce fragmentation.
Takes advantage of automatic segment space management, eliminating the need for manual space
management tasks.
Increases block-packing compared to the older model, reducing the overall number of buffer gets and thereby
improving runtime performance.
Makes best use of the restricted number of raw devices available in Oracle Real Applications Cluster (Oracle
RAC) and other environments, where every tablespace requires its own raw device.
Maximizes usefulness of wide disk stripe configurations.
Oracle Applications Tablespace Model (OATM):
Types & Contents
Object type Tablespace Type
Index Organized Transaction_tables
Tables
Global Temporary The database automatically always creates these objects in the TEMP tablespace.
Tables
Advanced Queuing Advanced Queuing/AQ
tables
Materialized View Summary
Materialized View Transaction_tables
Logs
Domain Indexes Transaction_indexes
Indexes With the exception of indexes on Tables classified as 'Transaction_Tables', all indexes will share the same
tablespace type as the base object (table/mv).
Media APPS_TS_MEDIA Multimedia objects, such as text, video, sound, graphics, and spatial data.
Archive APPS_TS_ARCHIVE Archive-purge-related objects
Undo UNDO Automatic Undo Management (AUM) tablespace. UNDO segments are identical to ROLLBACK
segments when AUM is enabled.
Temp TEMP Temporary tablespace for global temporary table, sorts, and hash joins.
System SYSTEM System tablespace.
Online Patching:
Pathway to Minimizing Planned Downtime
InfraOptimization
Online Patching:
References
Using the Online Patching Readiness Report in Oracle E-Business Suite Release 12.2 (Doc
ID 1531121.1)
Oracle E-Business Suite Developer's Guide Release 12.2 Part Number E22961-14
Oracle E-Business Suite Release 12.2: Online Patching FAQ (Doc ID 1583902.1)
InfraOptimization
source EBSapps.env run
adop -help
adop –examples
InfraOptimization
Online Patching:
Copy the Code, NOT the Data
• In 12.2 all patching operations are online by adop utility
– EBS will remain available to users during patching operations. In another word, critical business
operations were not interrupted by maintenance operations
– Downtime Redefined as "Cutover“, which is predictable: The time taken to bounce the Middle
Tiers
• Online patching uses
– EBS Edition Based Redefinition (EBR)
– Web Logic Server
• EBS uses both the file system and the database to store the code and data that make up the
application
– Code
• Stored both on the File system & In the Database
• Any code object changed in a patch is copied
– Data
• Stored both on the File system & In the Database
• Application data is NOT copied by a patch
Edition-Based Redefinition:
The key to Online Application Upgrade and Maintenance
Data changes are made safely by writing only to new columns or new tables not • Used by Online Users
seen by the old edition. An editioning view exposes a different projection of a table • Never changed by a
into each edition to allow each to see just its own columns. Patch
• Used by the Patching Tools
A cross-edition trigger propagates data changes made by the old edition into the • Changes do no affect the
new edition’s columns, or (in hot-rollover) vice-versa. running Application
InfraOptimization
AD Online Patching:
ADOP Patching Procedures
InfraOptimization
Adop Phase Scope
https://blogs.oracle.com/stevenChan/entry/webcast_replay_available_technical_preview
http://www.oracle.com/webfolder/technetwork/tutorials/obe/ebs/ebs121/atg/OnlinePa
tching.pdf
InfraOptimization
Monitoring:
Online Patching Session
InfraOptimization
1. Way of knowing history of online patching
Note:
Y denotes that the phase is done
N denotes that the phase has not been completed
X denotes that the phase is not applicable
R denotes that the phase is running (in progress)
F denotes that the phase has failed
P (is applicable only to APPLY phase) denotes at least one patch is already applied for the session id
C denotes that the status of this ADOP session has completed
Note: Numerical statuses are only relevant for the cutover phase...
These status values are updated when a step has completed, and are as follows:
cutover statuses
cutover_status='Y' 'COMPLETED'
cutover_status not in ('N','Y','X') and status='F' 'FAILED'
cutover_status='0' 'CUTOVER STARTED'
cutover_status='1' 'SERVICES SHUTDOWN COMPLETED'
cutover_status='3' 'DB CUTOVER COMPLETED'
cutover_status='D' 'FLIP SNAPSHOTS COMPLETED'
cutover_status='4' 'FS CUTOVER COMPLETED'
cutover_status='5' 'ADMIN STARTUP COMPLETED'
cutover_status='6' 'SERVICES STARTUP COMPLETED'
cutover_status='N' 'NOT STARTED'
cutover_status='X' 'NOT APPLICABLE'
2. Check the current status of the adop cycle
adop –status
usage
adop -status generates a summary report
adop -status <sessionID> generates a summary report for that session ID
adop -status -detail generates a detailed report
sqlplus <apps_schema_name>/<apps_Schema_password>
@$AD_TOP/sql/ADZDSHOWLOG.sql adzdshowlog.out file generated
@$AD_TOP/sql/ADZDSHOWDDLS.sql
or
set pagesize 200;
set linesize 160;
column adop_session_id format 999999999999;
column bug_number format a15;
column status format a15;
column applied_file_system_base format a23;
column patch_file_system_base format a23;
column adpatch_options format a15;
column node_name format a15;
column end_date format a15;
column clone_status format a15;
select adop_session_id, bug_number, status, applied_file_system_base, patch_file_system_base, adpatch_options, node_name, end_date,
clone_status
from ad_adop_session_patches
order by end_date desc;
Note: STATUS
N - Not Applied In the current node but applied in other nodes
R - Patch Application is going on.
H - Patch failed in the middle. (Hard Failure)
F - Patch failed in the middle but user tried to skip some failures.
S - Patch Application succeeded after skipping the failed jobs.
Y - Patch Application succeeded.
$COMMON_TOP/clone/FMW/logs/CLONE<timestamp>.error
$COMMON_TOP/clone/FMW/logs/CLONE<timestamp>.log
$INST_TOP/admin/log/clone directory
$APPLRGF/TXK/ohsCloneLog ( this captures cloning failures for OHS )
7. Ensure the following RUN and PATCH file system ORACLE_HOME(s) are registered in the inventory.xml
- 10.1.2 Oracle Home
- Oracle_EBS-app1 Oracle Home
- webtier Oracle Home
- oracle_common Oracle Home
8. Confirm that the FND_OAM_CONTEXT_FILES table now holds the correct data
Run the following script to determine if all context files have been registered in the database. The script should return a run and patch
context file for each application server and one for the database server.
select path,status,serial_number,version from fnd_oam_context_files where name not in ('TEMPLATE','METADATA') and (status is null or
status !='VALID');
InfraOptimization
Run the following script to determine the number/types of managed servers registered in the run and patch file systems. The script should
return an equal number of managed servers between the run and patch file systems
SELECT extractValue(XMLType(TEXT),'//host[@oa_var="s_hostname"]'),
extractValue(XMLType(TEXT),'//oacore_server_ports'),
extractValue(XMLType(TEXT),'//forms_server_ports'),
extractValue(XMLType(TEXT),'//oafm_server_ports'),
extractValue(XMLType(TEXT),'//forms-c4ws_server_ports'),
extractValue(XMLType(TEXT),'//oaea_server_ports')
from fnd_oam_context_files
where name not in ('TEMPLATE','METADATA')
and (status is null or status !='H')
and EXTRACTVALUE(XMLType(TEXT),'//file_edition_type')='run'
and CTX_TYPE = 'A';
SELECT extractValue(XMLType(TEXT),'//host[@oa_var="s_hostname"]'),
extractValue(XMLType(TEXT),'//oacore_server_ports'),
extractValue(XMLType(TEXT),'//forms_server_ports'),
extractValue(XMLType(TEXT),'//oafm_server_ports'),
extractValue(XMLType(TEXT),'//forms-c4ws_server_ports'),
extractValue(XMLType(TEXT),'//oaea_server_ports')
from fnd_oam_context_files
where name not in ('TEMPLATE','METADATA')
and (status is null or status !='H')
and EXTRACTVALUE(XMLType(TEXT),'//file_edition_type')='patch'
and CTX_TYPE = 'A';
InfraOptimization
InfraOptimization
Thank you
InfraOptimization