Professional Documents
Culture Documents
Configuring and Managing E-Business Application Tier for RAC (Doc ID 1311528.1)
Modified: 10-May-2013 Type: WHITE PAPER
This document describes how to configure and manage E-Business Suite Release 12 application tier database connections with Oracle Database 11g Release 2 RAC database.
Section 1 : Overview
Section 2 : Environment
Section 3 : Load Balancing Configurations
Section 4 : Failover Configurations
Section 5 : Configuring and managing database connection properties [ Includes changing ports and custom services. ]
References
The most current version of this note is available as My Oracle Support Knowledge <Document 1311528.1 >.
Section 1 : Overview
Oracle E-Business Suite running on Oracle Real Application Clusters (RAC) is proven technology to provide high availability and scalability. Once the basic E-Business Suite on RAC
environment has been deployed successfully, the user can use more advanced features on the Applications Tier.
This document covers both set up issues previously included in RAC documents, along with advanced configuration options such as load balancing, failover and services. It brings
together in one document all the information needed to successfully manage access to an E-Business RAC database.
Configuring Application Partitioning for both Forms and Java based modules.
This will provide various options to achieve load balancing in Forms and Java Based Modules by distributing the applications across RAC nodes.
Convention Meaning
Application Tier /Middle Tier Machines running Forms, Web, Concurrent Processing and other servers. Sometimes called middle tier.
CONTEXT_NAME The CONTEXT_NAME variable specifies the name of the Applications context that is used by AutoConfig. The default is <SID >_ <hostname >.
Full path to the Applications context file on the application tier or database tier. The default locations are as follows.
Application tier context file:
CONTEXT_FILE <INST_TOP>/appl/admin/CONTEXT_NAME.xml
Database tier context file:
<RDBMS ORACLE_HOME>/appsutil/<CONTEXT_NAME>.xml
Database_Name The name of the database which is equal to dbname initialization parameter value.
Monospace Text Represents command line text. Type such a command exactly as shown, excluding prompts such as '%'.
<> Text enclosed in angle brackets represents a variable. Substitute a value for the variable text. Do not type the angle brackets.
\ On UNIX, the backslash character can be entered at the end of a command line to indicate continuation of the command on the next line.
Section 2 : Environment
The logical configuration used to create this document is illustrated in the figure below. This documented was developed using Oracle E-Business Suite 12.1.1 Vision Rapid Install,
with database upgraded to 11.2.0.2.0 and converted to RAC.
https://support.oracle.com/epmos/faces/DocumentDisplay?_adf.ctrl-state=17ht1bbf0b_176&id=1311528.1 1/7
4/11/2014 Document 1311528.1
However, the processes detailed in this document apply to all R12.x releases.
Parallel Concurrent processing allows one or more concurrent managers to run on one or more nodes in a multi-node environment. You can take full advantage of Parallel
Concurrent Processing [PCP capabilities in a RAC environment.
You can define any set of concurrent managers on one or more nodes. For an example you can define two Standard Managers to spread across two nodes so that the load will be
distributed on two instances.
Assign a secondary node for the Internal Concurrent Manager to enable failover. The Internal Concurrent Manager can run on any node. It is responsible for starting all the
managers on all defined nodes.
Assign workshifts to the Internal Monitor managers and activate on the nodes which are a part of PCP configuration. The Internal Monitor is responsible for starting the
Internal Concurrent Manager on failover. Whenever there is failure in the Internal Concurrent Manager, the Internal Monitor tries to restart the manage on the same node,
otherwise the Internal Concurrent Manager will failover to the secondary node.
Define additional concurrent managers on the secondary nodes to enable load balancing.
You can also enable the fndreviver utility. To enable and configure fndreviver , refer to My Oracle Support Knowledge Document 466752.1.
For more details, refer Oracle E-Business Suite System Administrator's Guide - Configuration [ E12893-04]
1. Change the context variables s_cp_two task to <sid>_BALANCE alias in the context file.
2. To enable reviver modify s_cp_reviver to enabled.
3. Run Autoconfig and verify that the configuration files tnsnames.ora andlistener.ora files have FNDSM entries for all nodes.
4. Restart the application listener process on all application nodes.
5. Configure the Internal Monitor on all nodes. Logon to E-Business Suite as SYSADMIN, and choose the System Administrator Responsibility
1. Assign standard work shift. Logon to the E-Business Suite under System Administrator Responsibility ,
Concurrent--> Manager-->Define select internal monitor and click on Work shift button to assign
6. Verify that the Internal Monitor and Service Manager are running on all Concurrent Processing Nodes.
7. In Parallel Concurrent Processing implementation when the concurrent tier and database tier are different, then there are two options to provide high availability, depending
upon the value of the profile 'Concurrent:PCP Instance Check' .
1. If the value of the profile is "ON"then the concurrent manager will failover upon connected instance failure.
2. If the value of the profile is "OFF"then the Reviver process on concurrent tier tries to restart the Internal Concurrent Manager by connecting to the available database
node.
8. Define the secondary node for the required managers to enable failover by going to
Concurrent -> Manager -> Define -> select the manager Add the primary and secondary nodes
9. Define the standard managers to run in parallel on other nodes. For an example on the two node Concurrent Processing environment, create a standard manager with
multiple work shifts and assign the primary node as second node and then start that manager. Verify that the standard manager processes are running on the second node.
10. Configure the environment variables to allow concurrent manager logs to a shared location to all concurrent manager nodes and apps writable by owner of the oracle
database.
1. Set the $APPLCSF environment variable on all the CP nodes to a directory on a shared file system.
2. These changes need only be done on the concurrent manager tiers.
3.2 Configuring Application Partitioning for both forms and Java based modules.
https://support.oracle.com/epmos/faces/DocumentDisplay?_adf.ctrl-state=17ht1bbf0b_176&id=1311528.1 2/7
4/11/2014 Document 1311528.1
3.2 Configuring Application Partitioning for both forms and Java based modules.
E-Business Suite Release 12 has a large numbers of forms and java based modules. Load Balancing of Forms and Java Based Modules are more and more required and useful
especially when using Real Application Clusters. There are various methods to achieve load balancing of these components. This document covers the most common scenarios.
3.2.1 Implementing server side Load Balancing using Database connections for Forms Applications.
3.2.2 Configuring Hardware or WebCache as Load Balancer for Forms Applications
3.2.3 Application Partioning for Forms Applications.
3.2.4 Implementing Load Balancing for Java Based Modules
3.2.5 Mulltiple JVMs deployment for Load Balancing.
3.2.1 Implementing server side load balancing using Database connections for Forms Applications.
The E-Business Suite contains a large number of forms applications which connect to the database using net descriptors created by AutoConfig. In the Real Application Cluster
environment these connections can be load balanced across database instances using server-side load balancing. The advantage of database connection load balancing is it
improves performance by balancing the number of connections among various instances for the same service. The disadvantage of just using server-side load balancing is the
possibility of reduced performance due to database blocks being repeatedly pinged between database instances via the cluster interconnect.
Autoconfig creates net descriptors [tnsnames.ora aliases] with balance, instance and failover aliases.
1. Run the context editorthrough Oracle Applications Manager and set the value of Tools OH TWO_TASK (s_tools_twotask) to point to the <database_name>_balance
alias generated in thetnsnames.ora file.
2. Run Autoconfig.
3. Stop and start the Application Services.
Note : In this load balanced environment, if the instance to which forms session is connected goes down, you will receive an FRM-92100 error. You need to close the forms window
and login again to connect to any of the surviving instances.
3.2.2 Configuring Load Balancers for Forms and Self Service Applications
There are many options to configure the Load Balancing in Forms and Self Service application. For detailed configurations, refer to My Support Knowledge Document 380486.1
Installing And Configuring WebCache 10g and E-Business Suite 12 and My Support Knowledge Document 380489.1 Using Load-Balancers with Oracle E-Business Suite Release 12
Functional load balancing is an effective way to take the advantage of RAC databases. Group the end-users or responsibilities and assign to instances on the basis of access type so
that it reduces cache pinging between instances.
In order to direct certain classes of responsibilities and users to a particular instance in the Real Applications Cluster, set the Applications profile "Database Instance" to a specific
instance, this can be set at the Responsibility and Application level.
For example you can direct all Human Resource Forms users to instance 1 and Purchasing Forms users to instance 2 using this profile option.
3.2.4 Implementing Server side load balancing for Self Service Applications
The self service applications requires a JDBC connection to interact with the database. The load balancing of JDBC connections is achieved by setting the parameter
"APPS_JDBC_URL" in the dbc file associated with Applications.
1. Run context editor and set the value of Web OH TWO_TASK" and "Apps JDBC Connect Alias" to point to the <database_name>_balance alias generated in thetnsnames.ora
file.
2. Execute AutoConfig from $INST_TOP/admin/scripts/<context_name>/. Check the AutoConfig log file for errors.
3. Ensure that the dbc file locations $FND_TOP/secure/<context_name> and dbc file name is onlyservice_name.dbc and has the following APPS_JDBC_URL entry:
APPS_JDBC_URL=jdbc:oracle:thin:@(DESCRIPTION=(LOAD_BALANCE=YES)(FAILOVER=YES)
(ADDRESS_LIST=(ADDRESS=(PROTOCOL=tcp)(HOST=db_host1)(PORT=port1))
(ADDRESS=(PROTOCOL\=tcp)(HOST\=db_host2)(PORT=port2)))(CONNECT_DATA=
(SERVICE_NAME=<service_name>)))
4. Ensure that the value of profile option ?Applications Database Id? is set to <service_name> at site level.
Note: In this load-balanced environment, if the instance to which the self-service session is connected goes down, you will receive the java exception error. You need to
reconnect to the live instance by logging again to the application. If you want this failover to happen seamlessly set the parameter "FND_JDBC_USABLE_CHECK" to "true" in
dbc file located at $FND_TOP/secure. You may experience some performance degradation after setting this parameter value to "true"
To configure multiple JVMs for load balancing refer,to My Support Knowledge Document 362851.1 Guidelines to setup the JVM in Ebusiness Suite 11i and R12 .
4.Failover Configurations
This section describes how to configure Concurrent Processing ,Forms and Self Service applications for failover. It also explains how to distribute the load when failover event
occurs.
Failover in PCP is the process of migrating all the defined concurrent managers to a secondary node upon listener or node failure. You use the following test scenarios to verify that
https://support.oracle.com/epmos/faces/DocumentDisplay?_adf.ctrl-state=17ht1bbf0b_176&id=1311528.1 3/7
4/11/2014 Document 1311528.1
Failover in PCP is the process of migrating all the defined concurrent managers to a secondary node upon listener or node failure. You use the following test scenarios to verify that
the PCP configuration set up has been done properly. There are two possible cases, either listener failure or node failure. Note that node failure can be tested only if the node is not
pingable
When both nodes have listeners running and if ICM is down the internal monitor will restart the ICM on the same node. If the node is not pingable, then the ICM will failover to
secondary node.
Instance failover can be handled separately from middle tier failover when the concurrent processing nodes and database nodes are different. Set the Applications profile option
Concurrent:PCP Instance check. If this profile value is OFFthen the Internal Concurrent Manager restarts on the same node by connecting to a second instance when there is a
failure in connected instance.
On failover event, the survival nodes can be overloaded. There are certain features we can make use of to distribute the load.
One such feature in workshifts allows you to reduce the load on the secondary nodes. We can define fewer target processes than on the original node when there is a failover. The
number of failover target processes is entered as part of the standard Work Shift settings in the Service Instance Definition. When failover occurs, the ICM uses the Failover
Processes value in place of the normal running processes.
Transparent failover is not supported in Oracle Forms. The only option is for the form to be restarted. You can sets_tools_task to <service_BALANCE> in an application tier so that
all connections from that node will be direct to multiple instances. If there is an instance failure, all new connections go to surviving instances. Alternatively, if you are directing
forms to a specific instance, you can define a secondary instance [ or instances ] as a fallback, using _FO aliases with multiple instances. To generate failover aliases, complete the
following steps.
<ALTERNATE_SERVICE_INSTANCES oa_var="s_alt_service_instances">service:instance1,service:instance2,service:instance3</ALTERNATE_SERVICE_INSTANCES>
Then _FO aliases created with all the instances. Assign _FO aliases to certain responsibilities and users, they connect through an instance and failover when the connected instance
is down. For example _FO aliases, refer <service>_FO in Appendix A. And some of the low priority users to instance specific alias <sid>, so that there will not be a failover for
those users reduces the load on survival nodes.
Logon to E-Business Suite using sysadmin user. Click on system administrator responsibility and open Profile->system form. Select the responsibility (Example : Payables, Vision
Operations) or user and "Database Instance" profile and assign failover alias <service>_FO.
Transparent failover is not supported. If the database connection (s_weboh_twotask) is configured to <database_name>_balance then the new connection goes to the surviving
instance when the connected instance fails.
Create the failover aliases and instance specific aliases as mentioned in the above section, assign them to applications profile "Applications Database ID". so that some of
applications will not failover when that connected instance fails or is distributed to other instances. All stateless applications allow transparent failover, only stateful applications
needs to reconnect on an instance failure.
5 Configuring and managing database connection properties [ Includes changing ports and custom services. ]
This section describes configuring database port connection properties and creating and configuring custom services
5.1 Configuring Database Ports when SCAN Listener is used with E-Business Suite Release 12 (12.1.3 or above)
5.2 Creating and using custom services with E-Business Suite
5.1 Changing the Database ports in E-Business Suite Release 12 (12.1.3 or above)
This section explains the steps involved in changing database ports in E-Business Suite Release 12 (12.1.3 or above).
There are two options for listener the we can use in Oracle Database 11g Release 2.
If you are using the EBS database listener and would like to change the port, then follow the steps below on all database nodes.
1. Add the new database local listener with the new port.
1. Shutdown the existing listener.
2. Take the backup oflistener.ora andtnsnames.ora files.
3. Change the port inlistener.ora andtnsnames.ora to new port.
2. Register the instances with new listener.
3. Set the remote listener to register the instances across the nodes .
4. On the Database tier change the s_dbport in CONTEXT_FILE to new port and Run AutoConfig.
5. On Application tier then
1. Database Port s_dbport to <New Database Port>
2. Apps JDBC URLs_apps_jdbc_connect_descriptor (set this to an empty value " ")
3. Change the port to <New Database Port> intnsnames.ora for two task alias.
4. Run AutoConfig.
If you are using the SCAN Listener and want to change the port, perform the following steps
1. Shutdown the Database and Application Services.
2. Modify the SCAN Listener. From $CRS_HOME/bin execute following commands
https://support.oracle.com/epmos/faces/DocumentDisplay?_adf.ctrl-state=17ht1bbf0b_176&id=1311528.1 4/7
4/11/2014 Document 1311528.1
2. Modify the SCAN Listener. From $CRS_HOME/bin execute following commands
3. Change the database remote listener to register with the modified SCAN Listener for all instances.
4. Verify that all the instances are registered with SCAN Listeners
5. On the database tier change the s_scan_portcontext variable to new SCAN port number and run AutoConfig.
6. On the Application Tier use Oracle Applications Manager to change the following values on all APPL_TOP nodes:
1. Database Port s_dbport to <New SCAN Port>
2. Apps JDBC URL s_apps_jdbc_connect_descriptor (set this to an empty value " ")
3. Change the port to <New SCAN Port> in tnsnames.ora for two task alias.
4. Run AutoConfig.
Database service enables you to manage your application workloads as separate entities. You can control the processing resources that are allocated to each service and define
how these resources will be automatically reallocated during failures or planned outages. A single service can represent an application, multiple applications or a subset of a single
application. For example, E-Business Suite defines service for each responsibility such as Accounts Payable (AP), Customer Relationship Manager (CRM) and so on.
Using database services is made up of three parts: Creating the service, accessing the service and managing the service.
srvctl add service -d <DB_NAME> -s <name of the service> -r <preferred instance> -a <other instance> -P Basic -e Select -m BASIC -w 10 -z 5
-q TRUE
where
-d <db_unique_name> Unique name for the database
-s <service Service> name
-r "<preferred_list>" Comma separated list of preferred instances
-a "<available_list>" Comma separated list of available instances
-P {NONE | BASIC | PRECONNECT} TAF policy specification
-e <Failover type> Failover type (NONE, SESSION, or SELECT)
-m <Failover method> Failover method (NONE or BASIC)
-z <integer> Failover retries
-w <integer> Failover delay
-q AQ HA notifications AQ HA notifications (TRUE or FALSE)
For Example : Database Name = PROD and instances prod1 and prod2 in Two Node RAC and Service name is AP
#srvctl add service -d PROD -s APSer -r prod1 -a prod2 -P Basic -e Select -m BASIC -w 10 -z 5 -q TRUE
Middle tier applications access the service by specifying the name as part of the connection in TNS connect data.
1. Verify that the listener recognizes the service. Depending on the listener used either SCAN or non-SCAN
2. Construct a tns alias entry and add in $TNS_ADMIN/<sid>_<context_name>_ifile.ora. You need to add the preferred instance node and available instance node to the
connection descriptor. Example tns entry for service APser
APSer =(DESCRIPTION=(ADDRESS_LIST=
(FAILOVER=YES)(ADDRESS=(PROTOCOL=tcp)
(HOST=<node1>)(PORT=1521))
(ADDRESS=(PROTOCOL=tcp)(HOST=<node2>)(PORT=1521))
)(CONNECT_DATA=
(SERVICE_NAME=APSer) ) )
When the preferred instance is failed, then service is automatically relocated to an available instance. You can also manually relocate the service to balance services load. Once the
service has relocated automatically upon preferred instance failure, it will not fail back once the instance is available.
https://support.oracle.com/epmos/faces/DocumentDisplay?_adf.ctrl-state=17ht1bbf0b_176&id=1311528.1 5/7
4/11/2014 Document 1311528.1
service has relocated automatically upon preferred instance failure, it will not fail back once the instance is available.
Database services can be used with Forms, Java Based modules as well as with Parallel Concurrent Processing.
We can use the database services in Parallel Concurrent Processing. We assign the different services for different managers or requests so that the load will be balanced across the
instances.
There are two levels that can use services with PCP. One is at node level and the other manager/request level.
In a PCP environment multiple nodes will connect through different services which will have a different preferred instance. Each node in PCP will connect through a specific instance,
whenever service is relocated automatically due to preferred instance failure the ICM will failover to secondary node or restarts on the same node connecting to available instance
depends on the PCP configuration. The disadvantage is that, the relocated service will not fail back once the failed instance is up. When the preferred instance is relocated
fndreviver is enabled, the ICM will not failover to the other node, instead the concurrent manager will continue with the same service, but connecting to the second instance.
1. Create Services for example namely cpservice1 connect to instance1 and cpservice2 connect to instance2 using above command.
2. Assign s_cp_two task on first node to cpservice1 and second node to cpservice2.
Service Relocation :
To balance the services load on the instances we may need to relocate the service manually.
While running requests to distribute the load the service can be manually relocated to the second instance using the following command
When the connected instance goes down, the service can be automatically relocated to a second instance.
Note : The relocated services will NOT fail back automatically once the preferred instance comes up.
5.2.2.2 Forms and Self Service Applications using service based model
Using services with Forms is the same as usinf AutoConfig aliases, but the difference is that service aliases will be used. To use services, you should create the tns aliases and dbc
files as mentioned in the Configure Oracle Net to access the services step above. The advantages are that you can manage the workload effectively on the instances. In a RAC
environment Oracle clusterware recovers and balances service across instances based on the service definition. The disadvantage is that service will not fail back to a preferred
instance once it is up.
Define multiple services for example GL (General Ledger), AP (Accounts Payable) and so on and assign them to respective responsibilities and users.
1. Logon to E-Business Suite using sysadmin user. Click on system administrator responsibility and open Profile->system form. Select the responsibility (Example: Payables,
Vision Operations) or user and "Database Instance" profile and assign service (Example: Payable Service)
2. Transparent Failover will not happen, any failure in preferred instance of the service will be relocated to available instance application needs to reconnect. For self-service
applications, all stateless applications can failover transparently. Only stateful applications need to reconnect.
References
My Support Knowledge Document : 602899.1 Some More Facts On How to Activate Parallel Concurrent Processing
My Support Knowledge Document : 271090.1 Parallel Concurrent Processing Failover/Failback Expectations
My Support Knowledge Document : 752604.1 Failover Does Not Occur To The Secondary Node While The Primary Node Is Up
My Support Knowledge <Document : 729883.1 > How to Create a Second OPP Concurrent Manager in a Node Different Than The Primary Node
My Support Knowledge Document : 1057802.1 Best Practices for Performance for Concurrent Managers in E-Business Suite
My Support Knowledge Document :1129203.1 How to run a concurrent program against a specific RAC instance with PCP/RAC setup?
My Support Knowledge Document : 438921.1 Concurrent Manager Functionality Not Working And PCP Failover Takes Long Inspite of Enabling DCD With Database Server
My Support Knowledge Document : 762024.1 How To Ensure Load Balancing Of Concurrent Manager Processes In PCP-RAC Configuration
My Support Knowledge Document : 752604.1 Failover Does Not Occur To The Secondary Node While The Primary Node Is Up
My Support Knowledge Document : 380489.1 Using Load-Balancers with Oracle E-Business Suite Release 12
My Support Knowledge Document:362851.1 Guidelines to setup the JVM in Apps Ebusiness Suite 11i and R12
My Support Knowledge Document 417437.1 How to Prevent The Concurrent Managers to Start When Launching the Apps Middle Tier Startup Script "adstrtal.sh" ?
Appendix A
https://support.oracle.com/epmos/faces/DocumentDisplay?_adf.ctrl-state=17ht1bbf0b_176&id=1311528.1 6/7
4/11/2014 Document 1311528.1
Appendix A
ExampleTNSNAMES.ora enable _FO aliases
<service>_FO=
(DESCRIPTION_LIST=
(LOAD_BALANCE=NO)
(FAILOVER=YES)
(DESCRIPTION=
(ADDRESS=(PROTOCOL=tcp)(HOST=<host1>)(PORT=1521))
(CONNECT_DATA=
(SERVICE_NAME=<service>)
(INSTANCE_NAME=<instance1>)))
(DESCRIPTION=
(ADDRESS=(PROTOCOL=tcp)(HOST=)(PORT=1521))
(CONNECT_DATA=
(SERVICE_NAME=<service>)
(INSTANCE_NAME=<instance2>)))
)
)
Change Log
Date Description
05-Mar-2012
Initial Draft
REFERENCES
https://support.oracle.com/epmos/faces/DocumentDisplay?_adf.ctrl-state=17ht1bbf0b_176&id=1311528.1 7/7