TAKING ADVANTAGE OF

THE EMC CAPTIVA

ARCHITECTURE
Applying Best Practices to Optimize
Performance
Christopher Lund
EMC

Copyright 2011 EMC Corporation. All rights reserved.

Agenda
How InputAccel works
InputAccel 6.5 Benchmark Results
Tuning

InputAccel Server
Batches
InputAccel Database
Client Modules
Capture Workflow

Diagnosing Performance Issues

The EMC Captiva InputAccel and Dispatcher Version 6.5 Performance Sizing
and Tuning Guide which is available on PowerLink provided much of the data for
this presentation.

Copyright 2011 EMC Corporation. All rights reserved.

How InputAccel Works

A multi-machine capture application server
Server is the data tier (memory
mapped)

InputAccel System
Capture
Capture
Capture
Modules
Modules
Modules

Processing
Processing
Processing
Modules
Modules
Modules

Export
Export
Export
Modules
Modules
Modules

Server manages task queues

Server is multi-threaded
VBA execution is single-threaded

InputAccel
Servers

DB writes are queued, but singlethreaded

Server uses asynchronous I/O
Most work done from a thread
pool

WIP
&
Reports Tables

Copyright 2011 EMC Corporation. All rights reserved.

Clients are the executing tier

(where scaling comes from)

How InputAccel Works

An execution pipeline
Task queues are not FIFO

Execution Pipeline
m1

m2

m3

m4

Tasks are scheduled based on

Priority, then creation date

t1

B4

t2

B3

B4

t3

B2

B3

B4

t4

B1

B2

B3

B4

t5

A4

B1

B2

B3

A4

B1

B2

Assume short duration tasks

A2

A3

A4

B1

A1

A2

A3

A4

Work is pushed, no polling

A1

A2

A3

t6
t7
t8
t9

Copyright 2011 EMC Corporation. All rights reserved.

Recovery is through
reprocessing
Not a repository

Tasks may be prefetched

How InputAccel Works

Like a Petri net
A ProcessFlow defines the steps
and trigger levels.

Petri Net

Implicit fire when data is available

AA

BB

CC

There is no predefined execution

order
There is no end state IADone
triggered implies completed

Copyright 2011 EMC Corporation. All rights reserved.

InputAccel 6.5 Benchmark Results

One 8-core IA Server, over 300 Client Modules
In-house, ideal conditions, your mileage will vary
Performance similar to IA 6.0 SP1
Performance Level

Overall Task
Processing Rate
(tasks/hour)

Processing
Rate/CPU Core
(tasks/hour)

Avg. CPU
Utilization/C
PU Core

50 active batches
w/reporting disabled

2,672,007

324,001

67%

1000 active batches

w/reporting disabled

1,990,892

248,862

53%

1000 active batches

w/reporting enabled

1,384,910

173,114

32%

VMware ESX 4.x degrades throughput by approx 27%

Copyright 2011 EMC Corporation. All rights reserved.

Tuning InputAccel Server

It is a server application. It will use all available resources
Recommendation: use Windows 2008 R2 for best performance
CPU
ias.exe is multi-threaded
Recommendation: use at least a 4-core CPU for optimum throughput, 8+ is
better

RAM
Recommendation: 4-8 GB RAM, no less than 4 GB
ias.exe is a 32-bit app, so 4 GB address space max
BatchMaxAddressSpaceK controls how much RAM IAS uses
1.5 GB is the default
Set to 2 to 2.5 GB on 32-bit Windows with /3G option
Set to 3 to 3.5 GB on 64-bit Windows

Only as many batches as will fit in BatchMaxAddressSpaceK are kept in RAM,

when there are more, swapping occurs
Have your working set of batches small enough to all fit in RAM
Delete batches when you are done with them

Copyright 2011 EMC Corporation. All rights reserved.

Tuning InputAccel Server

Disk
Used heavily for batches and processes
RAID 1+0 is best RAID 5 usually is not fast enough and is not recommended

Recommendation: use a caching controller with Read Ahead and Write Back

SAN is OK, NAS not recommended

Turn off anti-virus scanning on IAS folder
IAS folder should be on a dedicated disk drive or array so it is not shared by other
programs or the Windows swap file
#2 cause of slow performance: a slow hard drive

Network
InputAccel has a chatty protocol between client modules and the InputAccel Server
For best performance, client modules, the InputAccel Server, Administration Console, and
DB should be on the same sub-net
WANs usually have low bandwidth with high latency, which may make it unsuitable

Connecting client modules to InputAccel Server via a WAN is doable with adequate performance depends on
the client modules, # batches on IA server, and IPP
InputAccel Server and IADB should not be connected by a WAN

Copyright 2011 EMC Corporation. All rights reserved.

Tuning Batches
Batch size (IAB file)
5 20 MB is ideal
< 100 MB is OK
> 100 MB not recommended, but is allowed
10 100 pages per batch recommended
1000 pages per batch degrades throughput by 10% or
more
< 10 pages per batch leads to too many small batches and
too much batch swapping

Copyright 2011 EMC Corporation. All rights reserved.

Tuning Batches
Number of Batches

Throughput is best when the working set of batches fit in RAM

Typically 50 500 fit depending on batch size and
BatchMaxAddressSpaceK

When there are more batches than fit in RAM, IA Server swaps
batches in/out as needed
Swapping decreases IA Server throughput

Up to 9,000 idle batches is possible with adequate performance

Idle means that all tasks within the batch have been processed and
finished

IA Server startup is slower with 1000s of batches because IAS

must load all batches into RAM to extract data
After startup they are swapped to disk and do not consume too many
resources

#1 cause of slow performance: too many active batches causing

excessive swapping

Copyright 2011 EMC Corporation. All rights reserved.

10

Tuning InputAccel Database

System Requirements
InputAccel Server requires a database to run
Only MSSQL is supported, versions 2005 and 2008
Express supported only for low volume and no IA Reporting
Standard or Enterprise recommended for medium to high volume

IADB stores:

IA configuration data
Reporting data on completed batches/tasks
Work In Progress (WIP) status
Web services data

Copyright 2011 EMC Corporation. All rights reserved.

11

Tuning InputAccel Database

System Requirements
Recommendation: 64-bit MSSQL server for best performance
2-4 CPU cores
4-8 GB RAM
RAID 1+0 with read/write caching controller w/ fast disks (15k RPM)

InputAccel Server requires fast, uninterrupted access to the DB

If the DB goes offline, IA Server pauses

Recommendation: put the DB and InputAccel Server on the

same low-latency, high-bandwidth subnet
Reporting decreases the InputAccel Server throughput by about
10-30%, although it can be more

Copyright 2011 EMC Corporation. All rights reserved.

12

Tuning InputAccel Database

Data Volume and Rates
IADB File Size and Growth Rates
Configuration data - typically < 100 MB
WIP data - 1 MB # batches
WIP is transient and grows and shrinks as needed

Error/Warning Log data typically negligible and can be purged as needed

Reporting data
All reporting log rules off 0 MB
Some or all reporting log rules are on

Typically data grows 100 MB 3 GB per hour

Shrinks only when purged

But
Growth rate depends on page volume and which log rules are on
Overall size depends on # days of data retained

Audit Log data

All audit log rules off 0 MB
All audit logs rules on 2 the growth rate of Reporting data

Copyright 2011 EMC Corporation. All rights reserved.

13

Tuning InputAccel Database

Data Volume and Rates
InputAccel Database transaction rates
Log Rules Enabled

Estimated Transaction Rate

None

IAS Tasks / Hour 0.075

Reports only (no Audit)

IAS Tasks / Hour 2.5

Audit only (no Reports)

IAS Tasks / Hour 5.0

Reports + Audit

IAS Tasks / Hour 7.5

Copyright 2011 EMC Corporation. All rights reserved.

14

Tuning InputAccel Database

Improving Performance
Defragment and rebuild indexes
up_ReorganizeIndex defragments all indexes
up_RebuildIndex rebuilds all indexes

Purge reporting and auditing data

Reporting and Auditing tables grow continuously
You must schedule purges via the Admin Console

Recommendation: generate reports during non-peak hours

Generating Reports runs complex queries that place a heavy load on
IADB and MSSQL

Store MSSQL transaction logs and data files on separate hard

drives/controllers

Copyright 2011 EMC Corporation. All rights reserved.

15

Tuning Client Modules

Each client module has its own unique tuning characteristics
Example: val2xml slows as more IA Values are exported and is slower when triggered at
level 1 than level 7
See EMC Captiva InputAccel and Dispatcher Version 6.5 Performance Sizing and
Tuning Guide for details

Parameters on the client machines can be modified to optimize performance

Stored in settings.ini located in %ALLUSERSPROFILE%\EMC\InputAccel
PrefetchDefault (default = 2) the number of additional tasks the InputAccel Server
sends to each client module
FileCacheSize (default = 8)
CacheSize (default = 1,048,576)
CacheCount (default = 200,000), previously 20,000 the number of files and IA Values
the client module caches
IAClientDebug (default = 0) set to 1 to capture the debug log iaclient.log in
%ALLUSERSPROFILE%\EMC\InputAccel (previously was created in C:\)

Copyright 2011 EMC Corporation. All rights reserved.

16

Tuning Client Modules

CPU intensive client modules like NuanceOCR &
ImageEnhancement
Require fast CPUs
Run one instance for each CPU core.
e.g. 4 CPU cores, run 4 instances of NuanceOCR

Non-CPU intensive client modules like val2xml & Documentum

Export
Performance is limited by other resources (disk, network)
Run at least one instance for each CPU core and possibly more

Copyright 2011 EMC Corporation. All rights reserved.

17

Tuning Client Modules

Using InputAccel over a WAN
InputAccel Server, Database and Administration Console
Require high-speed, low-latency connections
Must be on the same LAN as each other

Unattended client modules

Often require high-speed, low-latency connections for best throughput
Generally do not need to be remote from the InputAccel Server
Should be on the same LAN as the InputAccel Server

Attended client modules

Performance varies by environment and module
Detailed guidance is in the EMC Captiva InputAccel and Dispatcher
Version 6.5 Performance Sizing and Tuning Guide
Recommendation: do benchmark testing to ensure adequate performance

Copyright 2011 EMC Corporation. All rights reserved.

18

Tuning Client Modules

Using InputAccel over a WAN
Attended client module details
ScanPlus
Not recommended, but may perform acceptably with scanner hardware compression or small
scanned images
Works best when bandwidth is 50 Mbps and round trip latency 25 ms

IndexPlus
Generally performs well over a WAN (except for thumbnail display)
Displaying the batch list takes about 25% longer
Works best when bandwidth is 1.5 Mbps and round-trip latency 50 ms

Dispatcher Classification Edit and Dispatcher Validation

Should not be used over a WAN

Recommendation: consider using VMware View or Citrix for remote operators

Module executes on the LAN, screen display is over the WAN
Supports remote scanning
ScanPlus is on the LAN, the scanner is on the remote machine
Use scanner hardware compression for best results

Maximizes InputAccel Server-to-client module throughput

EMC OnDemand uses VMware View

Copyright 2011 EMC Corporation. All rights reserved.

19

Tuning Capture Workflow

Trigger Levels
Triggering at level 0 or 1
Usually gives better throughput than level 7
The tasks within a batch can be distributed among many client modules
Which results in faster end-to-end processing of any single batch

Triggering at level 7
Is less work for InputAccel Server as it has fewer tasks to manage
Under some circumstances may provide better overall throughput at the
expense that any single batch may take more time to process

Unsupported: accessing external resources in IPP VBA code

VBA execution with InputAccel Server is single-threaded

The external resource may be slow or not present
If InputAccel Server needs to wait for the resource, all other tasks block
Put more complicated custom code logic on the client through the .NET Code
module or client scripting

Copyright 2011 EMC Corporation. All rights reserved.

20

Diagnosing Performance Issues

IA Server Performance Counter

Batches loaded and loads/second

Connections
Disk bytes written & read/second
VBA calls/second & queue length
Processing Message Count
Network bytes written & read/second
Packets send & received/second
Pending I/O (~ # of asynchronous sends in progress)
Event (db) queue length

Data Access Layer Performance Counters

Data Requests/second
% Load Factor
Avg. Execution Time Millisec
Current connection count

Copyright 2011 EMC Corporation. All rights reserved.

21

EMC OnDemand
Captiva Instant Cloud Implementation

Copyright 2011 EMC Corporation. All rights reserved.

22

IIG Applications on EMC OnDemand

ENTERPRISE
CAPTURE

CONTENT
MANAGEMENT

CUSTOMER
COMMUNICATIONS

CASE
MANAGEMENT

Captiva

Documentum
ECM

Document
Sciences

Documentum
xCP

INFORMATION

GOVERNANCE
SourceOne

Cloud Management
Virtualization and Security
Storage
Networks

Copyright 2011 EMC Corporation. All rights reserved.

23

Summary/Key Takeaways
Focus on system throughput, not per-task time
Use high-speed hardware disk drives and networks
Minimize disk I/O where possible
Keep active batches in memory
Avoid excessive reporting

Parallelize multi-core CPUs and task granularity

Use performance counters to find bottlenecks

Copyright 2011 EMC Corporation. All rights reserved.

24

Q&A

Chris Lund
chris.lund@emc.com
(858) 320-1215

Copyright 2011 EMC Corporation. All rights reserved.

25

Learn More About EMC Captiva

Go to: www.EMC.com/Captiva

Copyright 2011 EMC Corporation. All rights reserved.

26

Captiva @ Momentum 2011

Thursday
10:00 AM

AP Automation: Best Practices for Capturing and

Integrating Paper Invoices into your Accounts Payable
Processes

Copyright 2011 EMC Corporation. All rights reserved.

Galileo 705

27

GET SOCIAL

DOWNLOAD
Apple: http://bit.ly/MMTM11
Other: http://bit.ly/MMTMemc

Come to the Momentum Lounge today

to play, win, learn and more.
Copyright 2011 EMC Corporation. All rights reserved.

28

 Copyright 2011 EMC Corporation. All rights reserved.

29

THANK YOU

Copyright 2011 EMC Corporation. All rights reserved.

30