Qualogy

2014-08-29
M. Schildmeijer

Whitepaper
Oracle Exalogic FMW Optimization

Inhoudsopgave
1. Preface ................................................................................................................................. 3
2. WebLogic Domain Level ...................................................................................................... 4
2.1 Domain Enhancements .................................................................................................. 4
2.2 JDBC SDP enhancement .............................................................................................. 4
2.3 Using AdminServer VIPS for floating IP ......................................................................... 5
2.4 WebLogic Admin channel over InfiniBand private net .................................................... 5
2.4.1

HTTP Client channel for AdminServer ................................................................ 5

2.4.2

T3 Client channels .............................................................................................. 6

2.5 Using Embedded Coherence (Several FMW products, SOA suite, IDM) ....................... 7
2.5.1 Configuring Coherence Caches for Dehydrations .................................................... 7
2.5.2 Enabling the property ............................................................................................... 7
2.5.3 Update SOA JVM settings ....................................................................................... 8
2.5.4 Overview of tuning settings ...................................................................................... 8
2.6 Cluster replications ....................................................................................................... 10
2.7. Configure JMS for Exalogic ......................................................................................... 12
2.7.1 JDBC Store ........................................................................................................... 12
2.7.2 JMS and Exalogic .................................................................................................. 12
2.8 Other Considerations .................................................................................................... 13
2.8.1 Hugepages and Transparent HugePages .............................................................. 13
3. Oracle Traffic Director enhancements ................................................................................ 14

1. Preface
For each WebLogic / FMW environment running on Exalogic, some extra optimization work
needs to be done in order to get the biggest benefit in terms of resources and performance
you can get when combining an Exalogic with an Oracle FMW. This whitepaper describes
the different configurations to optimize.
This whitepaper focuses on:

Java JVM/WebLogic/Exalogic optimizations

Domain Level enhancements such as: Cluster replication channels, SDP
enhancements
JDBC SDP enhancements
Domain specific enhancements
JMS & TLogs in database (after optimization)
SOA Suite specific optimizations
Cluster channel session replications
Coherence WKA over InfiniBand
Separate SOA traffic from Admin traffic
HTTP
Use of Oracle Traffic Director or use MNs Apache HTTP Proxy
SSL termination and offload

2. WebLogic Domain Level

2.1 Domain Enhancements
On a domain level, the next options have to be set:

-Dweblogic.ScatteredReadsEnabled=true Better I/O in high network volume

environments ( OSB, Webcenter Portal)
-Dweblogic.GatheredWritesEnabled=true
-Dweblogic.replication.enableLazyDeserialization=true

To be set in setDomainEnv
# Enable Java Exalogic optimizations
EXTRA_JAVA_PROPERTIES="${EXTRA_JAVA_PROPERTIES}
-Xlargepages:exitOnFailure=false -Doracle.xdkjava.exalogic.optimization=true
-Dweblogic.ScatteredReadsEnabled=true
-Dweblogic.GatheredWritesEnabled=true
-Dweblogic.replication.enableLazyDeserialization=true"
export EXTRA_JAVA_PROPERTIES

2.2 JDBC SDP enhancement

Besides configuring JDBC and GridLink for RAC to use InfiniBand, DB connections must use
the SDP protocol to improve performance:
Complete the following steps:
1. Ensure that you have created the Grid Link Data Sources for the JDBC connectivity
on both Exadata nodes
2. The console automatically generates the complete JDBC URL, as shown in the
following example:
jdbc:oracle:thin:@(DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=<>)(PORT=1521))(C
ONNECT_DATA=(SERVICE_NAME=<DB Service>)))

3. In the JDBC URL, replace TCP protocol with SDP protocol. For example:
jdbc:oracle:thin:@(DESCRIPTION=(ADDRESS=(PROTOCOL=sdp)(HOST=<>)(PORT=1521))(C
ONNECT_DATA=(SERVICE_NAME=<DB Service>)))

4. Manually add the system property -Djava.net.preferIPv4Stack=true to the

startWebLogic.sh script. Locate the following line in the startWebLogic.sh script:
a. . ${DOMAIN_HOME}/bin/setDomainEnv.sh $*
b. Add the following property immediately after the above entry:
JAVA_OPTIONS="${JAVA_OPTIONS} -Djava.net.preferIPv4Stack=true
-Doracle.net.SDP=true"

By having all these enhanced features, storing JMS artifacts in a JDBC store is more valid
than ever.

2.3 Using AdminServer VIPS for floating IP

The way of setting up an AdminServer VIP for HA and failover will be described in another
document. We assume that it is available for use.

2.4 WebLogic Admin channel over InfiniBand private net

The private InfiniBand fabric including WebLogic Clusters and Coherence clusters requires a
set of IP addresses for all WebLogic Managed Server-to-Administration Server traffic and for
cluster communication. These IP addresses are associated with the BOND0 interface.
Within Exalogic it is possible to use a private vnet that can be reserved for administrative
purposes, such as accessing the AdminServer and its consoles, internal cluster
communication and so on.
This can be easily achieved by setting all listen addresses to the IP that is bound to
InfiniBand

We do the same for all managed servers and their listen addresses
Trouble is that we cant access the AdminServer url over IPOIB. For this we create separate
channels.
Also for the managed servers, which have client access, we create channels to connect to
the outside world.
There are several channels to identify:
There are several channels for EoIB interfaces, such as:
AdminServer Network Channel
HTTP and T3 Network Channels
These are needed for internal resources, like AdminServers or to let managed servers
communicate with the outside world.
You must create two network channels for the Administration Server. The network channels
are necessary for routing HTTP traffic and T3 traffic (TCP-based protocol used by WebLogic
Server) coming in from an external data center via Ethernet over InfiniBand (EoIB). You must
create the following network channels:
2.4.1
1.
2.
3.
4.

HTTP Client channel for AdminServer

Log in to the WLS console and click Lock & Edit in the Change Center.
In the left pane of the Console, expand Environment, and then Servers.
In the Servers tab, click AdminServer (admin).
Select Protocols Channels. Click New.

5. Enter AdminHTTPClient as the name of the new network channel, select http as the
protocol, then click next.
6. Enter the following information in the Network Channel Addressing page:
Listen address: < the EOIB IP address >. This address is the floating IP
assigned to the Administration Server using the BOND<n> interface.
Listen port: 7001
7. Click next, and select Enabled on the Network Channel Properties page and click
Finish.
8. Activate these changes.
2.4.2 T3 Client channels
Repeat the same steps as above but choose T3 as a protocol.
Repeat these steps for all managed servers as well.
Example channels configured in a domain
AdminServer

Managed Servers

When started, you see the several interfaces which are listened to:
Sep 5, 2014 4:49:15 PM CEST> <Notice> <Server> <BEA-002613> <Channel "Default" is
now listening on 192.168.11.26:7001 for protocols iiop, t3, ldap, snmp, http.>
<Sep 5, 2014 4:49:15 PM CEST> <Notice> <Server> <BEA-002613> <Channel
"AdminChannel" is now listening on 10.16.160.21:7001 for protocols t3, http.>

And you still can use the normal AdminServer address to logon, either hostname or Admin
VIP configured.
When you configure Oracle Traffic Director, this should be configured using the IPoIB, in
every other setup you choose the EoIB, but then you miss all IB enhancements.

2.5 Using Embedded Coherence (Several FMW products, SOA

suite, IDM)
With optimizations in SOA Suite the process state can now reside in Oracle Coherence which is an in-memory data grid. Coherence uses direct memory access for replicating data.
We configure coherence using Unicast; therefore we should disable all multicast entries,
located in the setDomainEnv.
Furthermore, we should set all WKA addresses to use the IPoIB interface, by setting the
virtual hostname that is set for the IP address. This is typically the VIP address used for SOA
internal traffic (attached to IB interface)
-Dtangosol.coherence.wka1=SOAHOST1-PRIV-V1
-Dtangosol.coherence.wka2=SOAHOST2-PRIV-V1
-Dtangosol.coherence.localhost=SOAHOST1-PRIV-V1

SOAHOST1-PRIV-V1 is the virtual host name that maps to the virtual IP where WLS_SOA1
is listening (in SOAHOST1), and so on.
2.5.1 Configuring Coherence Caches for Dehydrations
Dehydrations take place in the SOA repository database, but you can use Coherence to
offload this kind of work. If you configure the BPEL engine with the CacheEnabled property,
the engine runs much fewer reads against database. Because many reads require locks and
version checks, eliminating them improves BPEL engine performance. Configuring Oracle
Coherence for dehydration requires the following steps:
2.5.2 Enabling the property
To enable the cache enabled property in SOA, follow these steps:
1. Log in to Oracle Enterprise Manager Fusion Middleware Control
2. Expand Farm_domain_name, SOA, and then right click soa-infra (server_name)
(either one of the servers).
3. Click SOA Administration, and then BPEL Properties.
4. Select More BPEL Configuration properties.
5. Enter CacheEnabled for the property QualityOfService property.
6. Select Apply.
Next, set the required server properties for using in-process coherence cache for
dehydration.
To enable the cache enabled property in SOA:
1. Log into the Administration Console
2. In the Domain Structure window, expand the Environment node.

3. Click Servers.
4. Click the name of the server (WLS_SOA1 or WLS_SOA2, which are represented as
hyperlinks) in the Name column of the table.
5. The settings page for the selected server appears.
6. Click Lock & Edit.
7. Click the Server Start tab.
8. In the Arguments field, add the following for all SOA managed servers
-Dbpel.cache.localStorage=true
-Dbpel.cache.threadCount=20
-Dbpel.cache.cubeInstance.sizeLimit=4g
-Dbpel.cache.invokeMessage.sizeLimit=2g
-Dbpel.cache.deliveryMessage.sizeLimit=2g
-Dbpel.cache.deliverySubscription.sizeLimit=2g

2.5.3 Update SOA JVM settings

To optimize behavior with local storage caches and improve performance, update the SOA
servers' start parameters to include the following flags:
-Djava.net.preferIPv4Stack=true
-Xlargepages:exitOnFailure=true
-Doracle.xdkjava.exalogic.optimization=true
-Xms<n>g -Xmx<n>g

( heapsize used)

2.5.4 Overview of tuning settings

Maybe an open door, but every setting you optimize must have a reason, so do not use all
these setting just like that. Do it one by one with a group of parameters, and test it. The
outcome of your analysis will give you a guideline of what is applicable and what is not. To
be set in the setDomainEnv, or better use a separate parameter file and reference it to the
setDomainEnv.

Java Optimizations:

EXTRA_JAVA_PROPERTIES="${EXTRA_JAVA_PROPERTIES}
-Xlargepages:exitOnFailure=false -Doracle.xdkjava.exalogic.optimization=true
-Dweblogic.ScatteredReadsEnabled=true
-Dweblogic.GatheredWritesEnabled=true
-Dweblogic.replication.enableLazyDeserialization=true"
export EXTRA_JAVA_PROPERTIES

Coherence Optimizations (after Java Optimizations are enabled)

# Used with InfiniBand

EXTRA_JAVA_PROPERTIES="${EXTRA_JAVA_PROPERTIES}
-Dsoa.archives.dir=${SOA_ORACLE_HOME}/soa
-Dsoa.oracle.home=${SOA_ORACLE_HOME}
-Dsoa.instance.home=${DOMAIN_HOME}
-Dtangosol.coherence.log=jdk
-Dtangosol.coherence.transport.reliable=imb

 InfiniBand enabling! IMB driver must be enabled for that. Only for Linux x86- 64 bit platforms.

-Djavax.xml.soap.MessageFactory=oracle.j2ee.ws.saaj.soap.MessageFactoryImpl
-Dweblogic.transaction.blocking.commit=true
-Dweblogic.transaction.blocking.rollback=true
-Djavax.net.ssl.trustStore=<location of your trust store>"
export EXTRA_JAVA_PROPERTIES

BPEL Optimizations. The main parameter is the bpel.cache.localStorage; set it to true

and then tune the other parameters to your needs, like cache sizes and worker
threads.

EXTRA_JAVA_PROPERTIES="${EXTRA_JAVA_PROPERTIES}
-Dbpel.cache.localStorage=true
-Dbpel.cache.threadCount=20
-Dbpel.cache.cubeInstance.sizeLimit=4g
-Dbpel.cache.invokeMessage.sizeLimit=2g
-Dbpel.cache.deliveryMessage.sizeLimit=2g
-Dbpel.cache.deliverySubscription.sizeLimit=2g"
export EXTRA_JAVA_PROPERTIES

Note that all the values are examples and can differ from your situation. Always measure,
analyze, test and decide!

2.6 Cluster replications

This applies only for stateful applications such as B2B Console, or SOA/BPM Composer.
To enable session replication enhancements for the SOA_Cluster:
1. Shut down the servers in the SOA_Cluster.
2. To set replication ports for a managed server, such as WLS_SOA1:
a. Under Domain Structure, click Environment and Servers. Click on the SOA
managed server, Configuration tab, Cluster tab.
b. In the Replication Ports field, enter a range of ports for configuring multiple
replication channels. For example, replication channels for managed servers
in SOA_Cluster can listen on ports starting from 8006 to 8009. To specify this
range of ports, enter 8006-8009.

3.

4.
5.
6.

10

c. Repeat these steps for all managed servers

Create a custom network channel for each managed server in the cluster:
Log on to the Admin Console in the Servers table; click the SOA managed server
instance.
a. Select Protocols, and then Channels, and click New.
b. Listen Address: SOAHOST1-PRIV-V1 ( (IPoIB Address )
Listen port: 8006 Remove the default external listen port.
Click next, and in the Network Channel Properties page, select Enabled and
Outbound Enabled, then click Finish.
Under the Network Channels, select ReplicationChannel, the network channel you
created for the WLS_SOA1 managed server.

7. Expand Advanced, and select Enable SDP Protocol and click Save.
You must repeat these steps to create a network channel, each for the remaining managed
servers in the SOA_Cluster cluster.
Next you should specify the replication channel to you cluster
1. Click the Replication tab.
2. In the Replication Channel field, ensure that ReplicationChannel is set as the name of
the channel to be used for replication traffic.
3. In the Advanced section, select the Enable One Way RMI for Replication option, and
click save.
4. Activate the changes and restart the managed servers.
5. Add the system property -Djava.net.preferIPv4Stack=true to the start parameters for
the WebLogic servers.
a. In the Domain Structure window, expand the Environment node and click
servers.
b. Click the name of the server.
c. Enter ReplicationChannel as the name of the new network channel and select
t3 as the protocol, then click next.
d. Click the Server Start tab. and add the following for the managed servers into
the Arguments field:
-Djava.net.preferIPv4Stack=true
Note that the domain should be restarted.
See logs, to see if it succeeded.
<Sep 5, 2014 5:53:16 PM CEST> <Notice> <Server> <BEA-002613> <Channel
"SOAReplChannel{4}" is now listening on 192.168.11.26:8009 for protocols t3,
CLUSTER-BROADCAST, http using SDP for I/O.>

11

2.7. Configure JMS for Exalogic

Although not a specific Exalogic enhancement, using all InfiniBand and SDP features storing
JMS Artifacts in a database is the preferred way to go.
2.7.1 JDBC Store
From here use the common methods to point JMS into JDBC store. However, keep the
following in mind:

Ensure that the JDBC data source used for the database work exists on the same
server instance as the JMS destinationthe transaction will still be two-phase, but it
will be handled with less network overhead.
Configure multiple services to share the same store if they will commonly be invoked
within the same transaction.
Do no use the XA driver for the JDBC Store, because the JMS infrastructure of
WebLogic itself is fully equipped for XA
JMS often executes direct database operations to invoke its store services within the
same transaction, so using a JDBC data source with Logging Last Resource (LLR)
might be a good option. It will do two-phase commit in a single local transaction DB
operation, this improves the overall transaction performance.

2.7.2 JMS and Exalogic

There are some enhancements specifically for Exalogic in doing message compression,
either persistency is file based or JDBC bases. The options you have to set are:

StoreMessageCompressionEnabled: Enabling Message Store Compression. For

Exalogic set this to true to use with the belonging options.
MessageCompressionOptions: Along with the option above, choose one of the
compression options in it. Note that compression is done by gzip, so you can choose
the best compression:
GZIP_DEFAULT_COMPRESSION
GZIP_BEST_COMPRESSION
GZIP_BEST_SPEED
LZF
PagingMessageCompressionEnabled setting. It's the JMS paging store to perform
message body compression on persistent and non-persistent messages.
JMS also makes use of the domain wide Exalogic optimizations by using concurrent queues
for pending messages. A buffer queue is used for incoming requests which increases
throughput by scheduling these requests and prevent locks.

12

2.8 Other Considerations

Some other considerations you could think of, either by determining if its applicable to your
situation, it may come out of some load or stress test scenario and analysis.
2.8.1 Hugepages and Transparent HugePages
Hugepages allocation can be good for applications which have to allocate large memory
blocks.
In your Exalogic VM infrastructure design, decide which VM will be used for such
applications like WebCenter Portal, and if they need it. In some situation, it can be necessary
to reserve a compute node for these environments instead of doing dynamic allocation, and
you might consider using Transparent HugePages for these.
For more info, see this fantastic article: http://www.qualogy.com/hugepages-for-weblogic/

13

3. Oracle Traffic Director enhancements

In almost every Enterprise Deployment Guide from a Fusion Middleware Product, Oracle
uses the Oracle WebTier as a controlling layer for you HTTP calls.
In Exalogic, a software loadbalancer is added, called Oracle Traffic Director. Depending on
your approach of Cloud infrastructure with one of these components

Network topology
Segregations of duties and tasks rolled out over several departments, or done by one
department overall? Who takes care of what? (Cloud Infrastructure team, Middleware
Team, all by one DevOps team)
Other solutions than those provided by Oracle (3rd party WebTiers, hardware
loadbalancers)

Besides the common steps to take in order to use Oracle Traffic Director, there are some
specific topics to address. These will be covered in a separate whitepaper.

14