Oracle Data Integrator

Hands-On Workshop
Version 2.2

Nguyen Tuan Khang

khang.nguyen@oracle.com (0913 008994)
Senior Solutions Consultant
Oracle Fusion Middleware
Oracle Vietnam

Copyright 2007
 Table of Contents
LAB 1: SOFTWARE INSTALLATION AND SETUP ................................................ 4
Download the Software (Pre-Requisite).......................................................................... 4
Install Oracle Data Integrator........................................................................................ 4
Install Oracle XE ........................................................................................................... 7
Setup the ODI Metadata Repositories............................................................................. 8
DATA INTEGRATION SCENARIO: PERMIT PROCESSING .............................. 19
LAB 2: CAPTURING METADATA........................................................................... 22
Configure Data Servers in the ODI Topology Manager................................................ 22
Define Data Models in the ODI Designer..................................................................... 27
LAB 3: SIMPLE DATA INTEGRATION .................................................................. 34
Create an Integration Project in the ODI Designer ...................................................... 34
Import the Knowledge Modules .................................................................................... 35
Create the Interface to Load the Applicant Data........................................................... 36
Define Basic Transformations ...................................................................................... 38
Define a Filter.............................................................................................................. 39
Execute the Interface.................................................................................................... 40
Monitor the Execution .................................................................................................. 41
LAB 4: HETEROGENEOUS DATA INTEGRATION AND VALIDATION .......... 45
Create the Interface to Load the Permit Data ............................................................... 45
Integrate Heterogeneous Sources ................................................................................. 47
Create Data Validation Rules....................................................................................... 50
Execute the Interface.................................................................................................... 53
Monitor the Execution .................................................................................................. 53
LAB 5: CHANGE DATA CAPTURE ......................................................................... 56
Import the Journalization Knowledge Module .............................................................. 56
Enable Journalization .................................................................................................. 56
Using Journalized Data in an Interface ........................................................................ 58
LAB 6: ORCHESTRATE INTERFACES INTO PACKAGES ................................. 61
Create the Permit Integration Package......................................................................... 61
Archive the Input File................................................................................................... 63
Execute the Package..................................................................................................... 65
LAB 7: SETUP AGENT .............................................................................................. 66
Setup ODI Parameters for Agent .................................................................................. 66
Create a Physical Agent ............................................................................................... 67
Create a Logical Agent................................................................................................. 69
Execute the Agent (use command line) ......................................................................... 70
LAB 8: CREATE AND SCHEDULE AN ODI SCENARIO ...................................... 71
Create an ODI Scenario............................................................................................... 71
Schedule a New Scenario within Oracle Data Integrator.............................................. 72
APPENDIX A: IMPORTING THE ODI REPOSITORY SCHEMAS...................... 77
APPENDIX B: RESETTING THE PERMIT APPLICATION SCENARIO............ 78

Page 2 of 79
Confidential
Before you begin
General notes:
1. All functions or actions performed by the student (for example, text to be entered or
buttons to click) are written in bold print.
2. When you see a reference to <hostname>, enter the name of the host for your assigned
system (for example, edrsrsp1). For this workshop, using localhost should work
correctly.
3. All references to the installation home directory of the Oracle Data Integrator are:
<install_dir>. (change this as per your install directory)
4. If a Welcome page appears as the first page of a wizard, then just click Next to advance to
the first step of the wizard.
5. The data used in this workshop are all fictional. Applicant names, SSNs, telephone
numbers, addresses and email addresses were all randomly created for training purposes.

Page 3 of 79
Confidential
Lab 1: Software Installation and Setup
Time estimate: 30 minutes (not including software download)

Download the Software (Pre-Requisite)

1. Go to the following web site:

http://www.oracle.com/technology/products/oracle-data-integrator/index.html

2. Follow the link to download:

Oracle Data Integrator (10.1.3.4.0) (odi_windows_x86_101340.zip)

3. We will use the Oracle XE database for the workshop. Go to the following URL:
http://www.oracle.com/technology/software/products/database/xe/index.html

4. Following the link to download:

Oracle Database 10g Express Edition (Western European) (OracleXE.exe)

Install Oracle Data Integrator

To install the Oracle Data Integrator, perform the following steps:

1. Unzip the odi_windows_x86_101340.zip file to some temporary directory <temp>.

2. From the <temp> directory where you unzipped the files, navigate to the sub-directory
\setup\windows and execute the setup.bat program to start the installer.

3. From the installation wizard, select the following options for each step:

Select a Product to Install: Oracle Data Integrator 10.1.3.4

Page 4 of 79
Confidential
 Installation Type: Complete

Oracle Home: ODIHome

Path: c:\oracle\product\10.1.3.4\oracledi

Page 5 of 79
Confidential
4. From the Summary Screen, Click Install.

5. From the End of Installation Screen, Click Exit.

Page 6 of 79
Confidential
Install Oracle XE

The Oracle Database 10g Express Edition (Oracle XE) is an entry-level, small-footprint
database based on the Oracle Database 10g Release 2 code base that's free to develop, deploy,
and distribute; fast to download; and simple to administer.

For this workshop, we will use Oracle XE as the database engine for ODI. To install Oracle
XE, perform the following steps:

1. Double-click the OracleXE.exe that was downloaded earlier to launch the installer.

2. Click Next.

3. Review and Accept the Licensing Agreement. Click Next.

4. Accept the Default Destination Location (c:\OracleXE). Click Next.

5. Enter a password for the SYS and SYSTEM accounts. Recommended to just use
oracle for the purposes of this workshop. Click Next.

6. From the Summary Screen, click Install.

7. From the InstallShield Wizard Complete window, check the Launch the Database
homepage. Then click Finish.

8. You should see the Oracle XE Database Login page from your browser. Do not close
the browser at this point.

Page 7 of 79
Confidential
Setup the ODI Metadata Repositories

Oracle Data Integrator uses 2 types of repositories:

The Master Repository is where all the information on the topology of resources,
security and version management of projects and data models is stored. Only one
master repository is required for typical Oracle Data Integrator installations.

The Work Repository is where the information on data models, projects, and their use
is stored. A work repository can be linked with only one master repository for version
management purposes.

These repositories are stored in a relational database accessible in client/server mode from the
different Oracle Data Integrator modules. For this workshop, we will use the Oracle XE
database to store these repositories.

Note: For this workshop, you may skip the creation of the master and work repositories
by running a script that will import the repository schemas into Oracle XE. If you wish
to do so, please refer to Appendix A: Importing the ODI Repository Schemas.

Setup Oracle Users

First, we need to create Oracle database users for the Master and Work Repositories.

1. From the Oracle XE Database Home Page, login using the system account with the
password you specified.

Page 8 of 79
Confidential
2. From the Database Home Page, select the Administration module, and from the menu
select Database Users > Create User.

3. From the Create Data User page, create a user called master and set the passwords to
master. Check the DBA role under User Privileges.

Page 9 of 79
Confidential
4. From the Manage Database Users page, click the Create button to create another user
called workrep1 and set the passwords to workrep1. Again, check the DBA role under
User Privileges.

5. Verify that the master and workrep1 users have been created as shown:

Page 10 of 79
Confidential
Install the Master Repository

To install the Master Repository on the Oracle XE database, perform the following steps:

1. From the Start Menu, go to Oracle > Oracle Data Integrator > Repository Management >
Master Repository Creation.

2. This will launch the Master Data Creation Wizard.

3. For the Driver, click the browse button and select the Oracle JDBC Driver from
the list. This should populate the text box with oracle.jdbc.driver.OracleDriver.

4. For the URL, click the browse button and select the
jdbc:oracle:thin:@<host>:<port>:<sid> URL from the list. In the populated text box,
replace <host> with localhost, <port> with 1521 and sid with XE. The resulting URL
should be:

jdbc:oracle:thin:@localhost:1521:XE

5. Enter master for the username and password.

6. In the Technology list, select Oracle.

7. Click the Test Connection button to verify that the database connection information is
correct.

Page 11 of 79
Confidential
8. Click the OK and OK again on the wizard to start the repository creation.

9. The Command Window will display log messages during the creation of the repository.
After successful repository creation, you should see the following:

10. Click OK to exit.

Page 12 of 79
Confidential
Access the Master Repository from the Topology Manager

1. Start the Oracle Data Integrator Topology Manager. From the Start Menu, go to Oracle >
Oracle Data Integrator > Topology Manager

2. The Security Repository Connection dialog box will prompt for login information. Click
on the New Connection icon to configure the Topology Manager to access the newly
created Master Repository.

3. Configure the Repository Connection as follows:

For the Oracle Data Integrator Connection use the default username
SUPERVISOR and password SUNOPSIS. This is the default admin user for the
master repository.

For the Database Connection use information you entered earlier for the master
repository connection (Username/Password: master).

Check the Default Connection option to make this the default.

Click Test to make sure the connection is valid and Click OK.

Page 13 of 79
Confidential
 4. Back in the Security Repository Connection, use the ODI Master Repository and
login with the user SUPERVISOR and password SUNOPSIS. This will launch the
Oracle Data Integrator Topology Manager connected to your Master Repository in the
Oracle XE database.

Install the Work Repository

To install the Work Repository on the Oracle XE database, perform the following steps:

1. Login to the ODI Topology Manager and click the Repositories tab (the 5th tab) from
the Topology Manager as shown:

2. Right-click on Work Repositories and select Insert Work Repository.

Page 14 of 79
Confidential
 3. On the Data Server Definition tab, enter the name WORKREP1 and select Oracle
from the Technology list. Under the connection, enter the username workrep1 and
password workrep1.

4. Click on the JDBC tab, enter oracle.jdbc.driver.OracleDriver for the JDBC Driver
and enter jdbc:oracle:thin:@localhost:1521:XE for the JDBC URL. You can also
use the browse button to select the values.

5. Click Test to make sure the connection is valid and Click OK. When prompted for an
Agent, select Local (No Agent).

6. Click OK.

7. The Work Repository dialog box will appear. For the ID, enter 0 and for the name,
enter WORKREP1 and click OK.

Page 15 of 79
Confidential
 Page 16 of 79
Confidential
Access the Work Repository from the ODI Designer

To login to the Work Repository, perform the following steps:

1. Start the Oracle Data Integrator Designer. From the Start Menu, go to Oracle >
Oracle Data Integrator > Designer.

2. The Oracle Data Integrator Login box will prompt for login information. Click on the
New Connection icon to configure the Designer to access the newly created Work
Repository.

3. Configure the Work Repository Connection as follows:

1. For the Oracle Data Integrator Connection use Login Name WORKREP1 with

Page 17 of 79
Confidential
 the default username SUPERVISOR and password SUNOPSIS. This is the
default admin user for the master repository.

2. For the Database Connection use information you entered earlier for the master
repository connection (Username/Password: master). You can also browse for the
JDBC driver and URL by clicking the browse icon on the right.

3. For the Work Repository Name, enter WORKREP1 (or you can browse for by
clicking the browse icon on the right.)

4. Check the Default Connection option to make this the default.

5. Click Test to make sure the connection is valid and Click OK.

4. Back in the Oracle Data Integrator Login, select WORKREP1 and login with the user
SUPERVISOR and password SUNOPSIS. This will launch the Oracle Data
Integrator Designer connected to your Work Repository.

You are now ready to use the Work Repository called WORKREP1 for the other sections of
this Workshop.

Page 18 of 79
Confidential
Data Integration Scenario: Permit Processing
Time estimate: 10 minutes

For this workshop, we will work on the integration requirements of a Permit Processing
application. The Permit Processing application is used for receiving and handling permit
requests. Aside from real-time permit processing, the Permit Processing application needs to
process permit requests in batches coming from various groups.

Multiple permit requests that include permit and applicant information are sent in a flat file.
The file typically contains more than 10000 requests including applicant information. This file
has to be extracted, transformed and loaded into the Permit Application tables. Background
check data coming from another relational database system will need to be merged to each
permit request as well.

Before the data gets loaded to the Permit Application, we need to make sure that the data is
valid (based on certain business validation rules). We also need to make sure that we only
process data that has changed from the background check system and that we archive the
permit files after processing it successfully. Lastly, there is a limited-time window involved
for processing, so we need to make sure that the data processing is done in the most efficient
manner.

----------------
----
----------------
----------------
----
----------------
----------------
Lo
---------------- ad

Permit Request
Flat File
Integration
BGCHECK
Transformation
ad
Lo Data Validation


CDC
Permit Application
Background RDBMS
Check
Data

Oracle Data Integrator

Metadata Repository

In this workshop, you will focus on building out this data integration scenario. You will use
the Oracle Data Integrator to do the following:

1. Capture the metadata information of these systems into ODIs metadata repository:
Permit Request flat-file
Permit Application RDBMS
Background Check RDBMS

2. Filter, transform and load applicant data into the Permit Application.

3. Merge permit requests with the background check data.

4. Create data validation rules and capture validation errors.

Page 19 of 79
Confidential
5. Use change-data-capture (CDC) to process changed data from the background check
system

6. Arrange these tasks in a data integration package with error notifications and file archival
after successful completion.

Before continuing, please perform the following tasks:

1. Open up the ODIWorkshop-MyFiles.zip file from the workshop DVD.

2. Unzip the file to your C:\ drive. This will create a directory called C:\MyFiles. This
includes the Permit Request flat file (permit_batch.csv in the CSV subdirectory) and
database scripts that will be utilized during the workshop.

3. Start a command prompt window. Go to Windows Start > Run and enter cmd.

4. From the command prompt, navigate to c:\MyFiles and enter the following commands:

sqlplus system/oracle

(Note: replace oracle with the password you specified during installation of
OracleXE.)

5. At the SQL Prompt, enter the following:

@workshopschema

This script creates the Permit Application and Background Check RDBMS schemas.

6. From the Command Prompt, enter the following:

LoadBGData

Page 20 of 79
Confidential
This command populates the background check RDBMS with data.

Page 21 of 79
Confidential
Lab 2: Capturing Metadata
Time estimate: 20 minutes

Before we can implement our data integration interfaces, it is essential to capture the various
sources and targets into a unified data integration platform. This lab shows how ODI can be
used to capture the metadata information for the Permit Application and Background Check
database schemas and the Permit Request Flat File into a central metadata repository.

Configure Data Servers in the ODI Topology Manager

The ODI Topology Manager is what we use to manage the physical and logical architecture of
the data sources. Each data source is configured as a data server belonging to a specific
technology. In this workshop, we will use Oracle database tables and a flat-file as our data
sources, so we need to create the data servers for these specific technologies.

1. Start the Oracle Data Integrator Topology Manager. From the Start Menu, go to
Oracle > Oracle Data Integrator > Topology Manager

2. Login using User SUPERVISOR and Password SUNOPSIS.

3. To define technologies, click the Physical Architecture tab (the 1st tab) from the
Topology Manager as shown:

Page 22 of 79
Confidential
Lets first capture the Permit Applications Oracle database that will be used as the target.

4. From the list of Technologies, right-click on the Oracle technology and select Insert
Data Server.

5. From the Data Server Definition Tab, enter the name PERMIT_APP. On the connection,
enter the username PERMIT_APP and password PERMIT_APP.

Page 23 of 79
Confidential
6. Click on the JDBC tab. For the Driver, click the browse button and select the
Oracle JDBC Driver from the list. This should populate the text box with
oracle.jdbc.driver.OracleDriver.

7. For the URL, enter jdbc:oracle:thin:@localhost:1521:XE.

8. Click OK.

9. In the Physical Schema dialog box, select PERMIT_APP for the Directory (Schema)
and PERMIT_APP_WORK for the Directory (Work Schema) as shown:

The Schema is where the actual tables are located, while the Work Schema is where ODI
will create temporary objects such as staging and integration tables.

Page 24 of 79
Confidential
10. Click on the Context tab. Click the new context button and type in
PERMIT_APP_TARGET as the Logical Schema.

This defines the schema in the Logical Architecture.

11. Click on OK.

Next, lets do the same steps for the Oracle database that contains background check
information. This will be used as a source.

12. Repeat steps 4-11 for the Oracle schema BGCHECK.

Name: BGCHECK
Connection User: BGCHECK
Connection Password: BGCHECK
JDBC Driver: oracle.jdbc.driver.OracleDriver
JDBC URL jdbc:oracle:thin:@localhost:1521:XE
Directory (Schema): BGCHECK
Directory (Work Schema): BGCHECK_WORK
Logical Schema BGCHECK_SRC

Important Note: The BGCHECK schema actually resides on the same database instance as
the PERMIT_APP. For this workshop, we are simulating that the BGCHECK is a totally
separate database server, which is why we are configuring it as a separate data server. In
real-life scenarios, if your database schemas belong to the same instance, the best practice is
to just use the same data server for these schemas.

Next, lets configure the Permit File source:

13. From the list of Technologies, right-click on the File and select Insert Data Server

Page 25 of 79
Confidential
14. From the Data Server Definition Tab, enter the name PERMIT_FILE.

15. Click on the JDBC tab. For the driver, click the icon and select Sunopsis File JDBC
Driver. For the URL, enter jdbc:snps:dbfile, as shown below:

16. Click OK.

17. In the Physical Schema dialog box, type c:\MyFiles\CSV for both Directory (Schema)
and Directory (Work Schema) as shown:

Page 26 of 79
Confidential
18. Click on the Context tab. Click the new context button and type in PERMIT_SRC as the
Logical Schema.

19. Click OK.

Define Data Models in the ODI Designer

After we configure the data servers, we can now define data models in ODI Designer. The
ODI Designer is what we use to define data models and integration projects that use the data
models. Before we can create integration projects, we need to be able to capture metadata
information from sources and targets and use this for defining the data models. This is the
process called Reverse Engineering, which we perform through the ODI Designer.

1. Login to the ODI Designer. Start the Oracle Data Integrator Designer. From the Start
Menu, go to Oracle > Oracle Data Integrator > Designer.

2. Login using User SUPERVISOR and Password SUNOPSIS.

Reverse Engineer the Permit Application Database

Lets start with Reverse Engineering the Permit Application target.

1. To define models, click the Models tab (the 2nd tab) from Designer as shown:

2. Click the new Model icon.

3. Enter PERMIT_APP_MODEL for the model name. Select Oracle as the Technology
and PERMIT_APP_TARGET as the Logical Schema.

Page 27 of 79
Confidential
4. Click the Reverse tab and select the context called Global.

5. Click the Selective Reverse tab and check the check boxes for Selective Reverse and
Objects to Reverse. Youll see the APPLICANTS, PERMITS and PERMIT_TYPES
tables from the list. Click the Reverse button to start the Reverse-Engineering Process.

Page 28 of 79
Confidential
6. You should now see the APPLICANTS, PERMITS and PERMIT_TYPES under your
data models with their corresponding columns and constraints.

Page 29 of 79
Confidential
Reverse Engineer the Background Check Database

Next, lets reverse engineer the Background Check database.

1. Click the new Model icon.

2. Enter BGCHECK_MODEL for the model name. Select Oracle as the Technology and
BGCHECK_SRC as the Logical Schema.

3. Repeat the Reverse-Engineering process done earlier to capture the BGCHECK table.

Reverse Engineer the Permit Request Flat File

Lastly, lets reverse engineer the Permit Request Flat File. The Permit Request Flat File,
located in c:\MyFiles\CSV\PermitBatch.csv, is a comma-delimited file that contains around
10,000 permit requests.

1. Click the new Model icon.

2. Enter PERMIT_SRC_MODEL for the model name. Select File as the Technology and
PERMIT_SRC as the Logical Schema.

Page 30 of 79
Confidential
Note: If you do not see the PERMIT_SRC in the Logical Schema combo box (just see
FILE_GENERIC_SRC), check the Topology again that you put the PERMIT_SRC in the File
technology

3. Click the Reverse tab and select the context called Global.

4. Click OK. (For Files, we do not perform Selective Reverse since each file is reverse-
engineered individually.)

5. From the Models window, right-click on PERMIT_SRC_MODEL and select Insert

Data Store.

Page 31 of 79
Confidential
6. Enter PERMIT_CSV for the name and click the Browse icon to locate the
permit_batch.csv file.

7. Click the Files tab and select Delimited for the File Format, 1 for the Heading and
comma (,) for the Other Field Separator:

Page 32 of 79
Confidential
8. Click the Columns tab and click the Reverse button to generate the column names based
on the file header as shown:

9. Click OK.

You have successfully captured the sources and target metadata for the data integration
scenarios in the next labs.

Page 33 of 79
Confidential
Lab 3: Simple Data Integration
Time estimate: 30 minutes

Now that the Permit Application, Background Check database and Permit Request File data
models have been defined, we can use them as sources or targets in our interfaces. For this
lab, we will create a simple data integration interface that reads applicant data from the Permit
Request file, performs basic transformation, and then stores it in the Permit Applications
applicant table.

Create an Integration Project in the ODI Designer

As mentioned earlier, the ODI Designer is what we use to define data models and integration
projects that use the data models. Now that data models are defined, we can proceed in using
the data models for interfaces in data integration projects. Lets start by creating a new
Project for this workshop.

1. Login to the ODI Designer.

2. To create a project, click the Projects tab (the 1st tab) from Designer as shown:

3. Click the new Project icon.

4. Enter Integration Project 1 for the project name.

Page 34 of 79
Confidential
5. You will now see an Integration Project 1 project in your project list. Click the + sign
on the left to expand and you will see a default folder called First Folder. Double-Click
the First Folder and rename this to Permit Integration as shown:

Import the Knowledge Modules

We will now import the knowledge modules that will be used for loading the Permit File
Source to the Permit Application.

Knowledge Modules are customizable integration strategy templates that can be

re-used to perform a specific set of tasks against a specific technology or set of
technologies. ODI comes out of the box with more than 100 Knowledge Modules
for the major database engines and application packages of the market.

We will use the following knowledge modules:

1. Loading Knowledge Module LKM File to SQL

2. Integration Knowledge Module IKM SQL Incremental Update
3. Check Knowledge Module CKM Oracle

To import the knowledge module:

1. Expand the Knowledge Modules folder under Integration Project 1 and you will see the
various types of knowledge modules.

2. To Import the Loading Knowledge Module called IKM File to SQL, Right Click on the
Loading (LKM) folder and select Import Knowledge Module.

3. From the Import Knowledge Modules File Import Directory, enter or browse for the
following path:

Page 35 of 79
Confidential
 c:\oracle\product\10.1.3.4\oracledi\impexp
(Replace with your ODI installation home directory if you used a different one.)

4. From the list of files(s) to import, select LKM File to SQL, as shown.

5. Repeat the same steps for the other knowledge modules listed above. You should see the
imported knowledge modules as shown:

Create the Interface to Load the Applicant Data

Now that the knowledge modules have been imported, we can use ODIs Declarative Design
framework in building the interface. With Declarative Design, you focus on what the rules
for your integration instead of the underlying technical aspect of the process. The imported
knowledge modules will take care of that for you.

Lets create the interface to load the applicant data from the Permit Data file:

1. Expand the Permit Integration folder, right-click on the Interfaces folder and select
Insert Interface.

2. For the Interface name, enter Load Applicant Data.

Page 36 of 79
Confidential
3. Click the Diagram tab.

4. You drag and drop data stores from models as sources and the target in this diagram. To
do this, select the Models tab in the left-navigation window (2nd tab from the bottom of
the left navigation window) to display the list of Data Models.

5. Drag the PERMIT_CSV data store from the PERMIT_SRC_MODEL and drop it on
the Source section.

6. Drag the APPLICANTS data store from the PERMIT_APP_MODEL and drop it on the
Target section.

Page 37 of 79
Confidential
7. You will be asked if you want to perform an automatic mapping. Select Yes.

8. The mapping is based on exact field names so all the fields except the APPLICANT_ID
will be automatically mapped.

(Note that you will see a warning for the Phone and SSN mappings, indicating that the
target column is smaller than the source column. This can be ignored.)

9. To manually map the APPLICANT_ID, click the empty mapping field of the
APPLICANT_ID column from the Target Data Store. This will bring up the
APPLICANT_IDs mapping implementation below (which is empty). Drag the Appl_Id
column from the PERMIT_CSV source node to the mapping implementation text area as
shown:

Define Basic Transformations

Now lets add some basic transformations. Well add a transformation to the first name and
last name so that the first letter is capitalized.

Page 38 of 79
Confidential
1. Click the mapping field of the APPLICANTS FIRST column from the Target Data
Store. This will bring up the mapping implementation below, which shows the
MAPPING as PER.First.

2. ODI is based on an E-L-T architecture that utilizes the database engines transformation
capabilities. For this transformation, we will use Oracles INITCAP function. To do this,
we need to specify to execute the transformation in the Staging Area rather than the
source. Select the option Execute on Staging Area below the mapping implementation.

3. Enter the mapping implementation as: INITCAP(PER.First) as shown below.

Optionally, you can click on the icon on the right to launch the expression editor and
select the INITCAP function under the list of String functions.

4. Repeat the same transformation for the last name column (APPLICANTS.LAST).

Define a Filter

Now lets add a filter to get the permits with permit ID greater than 4008000.

1. From the sources window, drag the Permit_ID column outside the PERMIT_CSV node
to create a filter.

2. Click on the filter icon to show the filter implementation expression below and enter the
expression PER.Permit_ID > 4008000.

Page 39 of 79
Confidential
Execute the Interface

Now were ready to fire off this interface.

1. Click on the Flow tab. The diagram shows you how the execution takes place.

Notice that only the source and target nodes are show. There is no separate
transformation engine. All the transformation and filtering is done at the target
database where the data is processed and loaded using a set-based operation rather
than processing it row-by-row in a separate engine. This approach yields much
higher performance and shows the benefits of the E-L-T Architecture.

2. Click on the source node and the target node to see the available options provided by the
knowledge modules.

Page 40 of 79
Confidential
 Notice that all we had to for this interface was drag the source to the target and specify
the transformations and filters, which are all part of the integration rules. The
knowledge modules take care of all the underlying technical details such as creating the
staging and error tables, committing, cleaning up temporary objects, and performing
flow controls, among others. These are the benefits of Declarative Design.

3. Click on the Execute button below.

4. Click OK in the execution dialog box. You will be informed that the session has started.
Click OK again.

Monitor the Execution

The ODI Operator is what we use to monitor the execution of sessions.

1. Start the ODI Operator by clicking the Operator icon from top menu in Designer.

You can also use the Start Menu: Oracle > Oracle Data Integrator > Operator.

2. Drill down on All Executions and find the Load Application Data session. Click and
expand the second Load Application Data step. You will see the loading, integration and
control steps implemented by the knowledge modules executed by ODI.

Page 41 of 79
Confidential
3. On the left hand window, scroll to the right to view the various statistics for each step.
Youll see that 18000 rows have been staged and 10000 rows have been inserted into the
target table.

4. Double-click Step 7: Integration Load Applicant Data - Insert flow into I$ table.
This is the step that integrates the source data in the staging area.

5. Click on the Description tab and examine the SQL generated with the transformations
and filters that you defined.

Page 42 of 79
Confidential
6. Lastly, go back to ODI Designer, click on the Diagram tab for the Load Applicant Data
interface and right click the Target Data Store - APPLICANTS header as shown:

7. Select Data to view the data that has been inserted:

Page 43 of 79
Confidential
Congratulations! You have successfully implemented an interface that loaded 10000
applicants from a flat-file into a target database.

Page 44 of 79
Confidential
Lab 4: Heterogeneous Data Integration and Validation
Time estimate: 30 minutes

We will now create the interface that populates the PERMIT table. The data will still come
from the permit file, but this time, we will need to do a heterogeneous join between the file
and the background check database table to get the results of the background check for the
applicant. We will also apply data validation rules to the data models so that only valid data
is stored in the Permit Application. Invalid data will be captured in an error table.

Create the Interface to Load the Permit Data

Lets create the interface to load the permit data from the Permit Data file:

1. Start and login to the ODI Designer, if it is not already running.

2. Expand the Permit Integration folder, right-click on the Interfaces folder and select
Insert Interface.

3. For the Interface name, enter Load Permit Data.

4. Click the Diagram tab.

5. Now once again, well drag the models as sources and targets. Select the Models tab in
the left-navigation window (2nd tab from the bottom of the left navigation window) to
display the list of Data Models.

Page 45 of 79
Confidential
6. Drag the PERMIT_CSV data store from the PERMIT_SRC_MODEL and drop it on
the Source section.

7. Drag the PERMITS data store from the PERMIT_APP_MODEL and drop it on the
Target section.

8. You will be asked if you want to perform an automatic mapping. Select Yes.

9. Like earlier, manually map the Appl_Id column from the PERMIT_CSV source to the
APPLICANT_ID target column.

10. Add the TO_DATE conversion function to the PER.Application_Date and

PER.Request_Date target columns to convert the string data into a date.

Make sure you specify to execute this transformation in the Staging Area.

11. For the LOAD_DATE column, use the Oracle SYSDATE function to record the current
date.

Page 46 of 79
Confidential
12. Like earlier, create a filter for Permit_ID using the expression PER.Permit_ID >
4008000.

13. You should now have something like:

Integrate Heterogeneous Sources

Now we need to get the background check data from another relational system to join to the
Permit File using the Social Security Number.

1. Since we are now using a database as a source, we need to import the appropriate
knowledge module. Go to the Projects tab, expand the Knowledge Modules folder and
right-click on the Loading (LKM) folder to import the knowledge module called: LKM
SQL to Oracle.

2. Switch back to the models tab and drag the BGCHECK data store from the
BGCHECK_MODEL and drop it on the Source section.

Page 47 of 79
Confidential
3. Drag the SSN column from PERMIT_CSV to the SSN column in BGCHECK to create
the join.

4. Map the BGCHECKs RESULT column from the source to the PERMITs
BACKGROUND_CHECK column in the target.

5. Map the BGCHECKs REASON column from the source to the PERMITs
BACKGROUND_CHECK_MSG column in the target.

6. We need to only accept background check data only for the current year. Create a filter
on the BGCHECKs DATE_CHECKED column using the expression:

BGCHECK.DATE_CHECKED > TO_DATE('01-JAN-07')

We also need to lookup the correct permit type codes from the Permit Applications
PERMIT_TYPES table because the permit data file only contains the permit type name. To
do this, we need to include the PERMIT_TYPES table as a source.

Page 48 of 79
Confidential
7. Drag the PERMIT_TYPES data store from the PERMIT_APP_MODEL and drop it on
the Source section.

8. You will be asked if you want to perform an automatic mapping. Select Yes. This will
automatically create the mapping for the PERMIT_TYPE_CODE.

9. Drag the Permit_Type column from PERMIT_CSV to the PERMIT_TYPE_NAME

column in PERMIT_TYPES to create the join.

10. You should now have the following:

Note:
- Because Technology File does not support all functions like RDBMS, make sure that
you set the Execute on for the following fields on Staging Area instead of
Source: APPLICATION_DATE; REQUEST_DATE; APPLICANT_ID; LOAD_DATE
- Make sure that you have used TO_DATE for APPLICATION_DATE, REQUEST_DATE
- You should use: PERMIT_ID = TO_CHAR(PER.Permit_ID) if you receive an mis-datatype
error

11. Click on the Flow tab and observe how and where the transformations and merges take
place:

Notice that the joins are done in the staging area. The sources get staged first and
then joined in the staging area for best performance.

Page 49 of 79
Confidential
 Notice that PERMIT_TYPES, although we brought it in as a source, is not staged like
the other sources. ODI detects that PERMIT_TYPES belongs to the same data server
as the PERMITS target and therefore it integrates this directly in the staging area.

Notice that the filter for BGCHECK occurs on the source since BGCHECK is a
relational source that can perform filtering logic. It will be more optimal to perform
filtering on the source when possible to minimize the data coming in. Since the
PERMIT_CSV is a file, it does not have an engine to perform filtering and so it
utilizes the staging area for applying the filters.

12. Click Apply to save the interface.

Create Data Validation Rules

Before we execute, lets add some validation rules on the Permit data model. These rules will
be enforced during execution time. Data that do not meet the rules will be rejected and
captured in an error table, where it can then be corrected and reprocessed.

1. Go to the Models tab and drill down on PERMITS and view its Constraints. There are 2
constraints there, the primary key and the foreign key to the APPLICANTS table. These
were captured during the reverse-engineering process.

We will now add 2 additional constraints that will contain our validation rules. Lets create a
rule for making sure that each permit application has a valid reason.

Page 50 of 79
Confidential
2. Right-click the Constraints folder and select Insert Condition.

3. Enter the following information:

Name: Reason is Required

Type: Oracle Data Integrator Constraint
Where: PERMITS.REQUEST_REASON IS NOT NULL
Optionally, use the Expression Editor to build the expression.
Hint: Use the Test Query icon to validate your expression.
Message: There has to be a reason

Note you can also create "NOT NULL" rules at the column level by checking the
"mandatory" check box. The advantage of utilizing conditions as shown above is that error
messages can be customized.

4. Click Ok.

Now lets create a rule to make sure only valid and completed background check results are
accepted.

5. Right-click the Constraints folder and select Insert Condition.

6. Enter the following information:

Name: Background Check Validation

Type: Oracle Data Integrator Constraint
Where: PERMITS.BACKGROUND_CHECK IN ('Pass','Fail')
Optionally, use the Expression Editor to build the expression.
Hint: Use the Test Query icon to validate your expression.
Message: Background check needs to be Pass or Fail

Page 51 of 79
Confidential
7. Click Ok.

8. Go back to the Load Permit Data Interface and click the Controls tab. Verify that your
validation rules are listed under constraints and that they are enabled.

Page 52 of 79
Confidential
Execute the Interface

1. Click on the Execute button below.

2. Click OK in the execution dialog box. You will be informed that the session has started.
Click OK again.

Monitor the Execution

1. Go to the ODI Operator by clicking the Operator icon from top menu in Designer.

2. Drill down on All Executions and find the Load Permit Data session. Click and expand
the second Load Permit Data step.

3. On the left hand window, scroll to the right to view the various statistics for each step.
Notice that 8066 rows loaded with errors caught for the PK and CK constraints that you
created.

4. Optional: Try to locate the step that contains the SQL logic for integrating the 3 data
sources.

5. Go back to ODI Designer. From the Models tab on the left, right-click on the PERMITS
data store from the PERMIT_APP_MODEL and select View Data.

Page 53 of 79
Confidential
6. You seed the data that was inserted as shown below:

7. To view the captured errors, right-click on the PERMITS data store from the
PERMIT_APP_MODEL and select Controls > Errors.

8. You should see the errors that have been captured and the corresponding error messages
as shown below:

Page 54 of 79
Confidential
You have successfully implemented an ODI interface that performs heterogonous data
integration and validation.

Page 55 of 79
Confidential
Lab 5: Change Data Capture
Time estimate: 30 minutes

ODI provides a Change Data Capture (CDC) framework to be able to identify and capture
changed data from the sources (inserted, updated, or deleted data) and use these changes in
integration processes. Changed Data Capture is performed by enabling journalization on data
models. Journalizing a model consists of setting up the infrastructure to capture the changes
(inserts, updates and deletes) made to the records of this model's data stores. Behind the
scenes, the CDC framework utilizes database triggers or log mining capabilities of the
database. These are implemented by the Journalization Knowledge Modules.

For this workshop, we will enable journalization on the RDBMS source, which is the
BGCHECK database table. This will enable us to only new or updated background check
information.

Import the Journalization Knowledge Module

1. Start and login to the ODI Designer, if it is not already running.

2. Go to the Projects tab, expand the Knowledge Modules folder and right-click on the
Journalization (JKM) folder to import the knowledge module called: JKM Oracle
Simple.

Enable Journalization

To enable journalization on the BGCHECK data store, perform the following steps:

1. Go to the Models tab and double-Click the BGCHECK_MODEL and click on the
Journalization tab.

2. For the Journalizing JKM, select JKM Oracle Simple.Integration Project 1

Page 56 of 79
Confidential
3. Expand the BGCHECK_MODEL, right-click on the BGCHECK data store and select
Change Data Capture > Add to CDC.

4. Next, right-click on the BGCHECK data store and select Change Data Capture >
Subscriber > Subscribe.

5. When asked to Select your Subscriber(s), enter SUNOPSIS and click the icon to add
the ODI subscriber to the list as shown. Click OK.

6. Click OK in the execution dialog box. You will be informed that the session has started.
Click OK again.

7. Right-click on the BGCHECK data store and select Change Data Capture > Start

Page 57 of 79
Confidential
 Journal.

8. When asked to Select your Subscriber(s), select SUNOPSIS from the list and click OK.

9. Click OK in the execution dialog box. You will be informed that the session has started.
Click OK again.

10. Go to the ODI Operator and verify that the 2 jobs for BGCHECK completed successfully.
Optionally, examine the steps under each job. These are the journalization setup work
performed by the Journalizing Knowledge Module (JKM).

Using Journalized Data in an Interface

1. Open the Load Permit Data Interface that you create earlier and click the Diagram tab.

2. You can preview the source data that will be loaded by right clicking on the BGCHECK
source node and selecting Data.

Page 58 of 79
Confidential
3. Notice the BGCHECK table contains several rows that have already been processed
earlier. This is because the interface is not set up to use the journalized data.

4. To use the journalized data in the interface, click on the BGCHECK source node header.
From the source configuration below the diagram, mark the Journalized Data only
option as shown below.

5. Notice that a new filter has been created on the BGCHECK source. Click the filter icon to
see the filter expression.

6. Try right clicking on the BGCHECK source node again and select Data. There are no
new rows to process.

7. Lets try adding some new rows. Start a command prompt and enter the following
command:

sqlplus BGDEMO/BGDEMO

8. At the SQL Prompt, enter the following:

Page 59 of 79
Confidential
 @c:\myfiles\newBGData.sql

9. This will insert 10 new background check results into the background check table.

10. Go back to Load Permit Data interface in ODI Designer and view the BGCHECK source
data again. This time you will see 10 new rows waiting to be processed.

11. Execute the interface and verify through the ODI Operator that only10 new rows have
been added and not the entire data set.

Page 60 of 79
Confidential
Lab 6: Orchestrate Interfaces into Packages
Time estimate: 30 minutes

Now, let's put together these 2 interfaces in a data integration package. A package is a
sequence of steps organized in an execution diagram. The steps can be interface executions,
variable assignments, procedure calls, file operations, web service calls, OS commands,
notifications and many other utilities. Packages can then be compiled as scenarios, which can
be invoked by external applications such as web service clients.

Create the Permit Integration Package

1. Go to ODI Designer and click on the Projects tab.

2. Expand the Permit Integration folder, right-click on the Packages folder and select
Insert Package.

3. For the Package name, enter Run Permit Integration.

4. Click the Diagram tab. You can drag and drop interfaces that you want to orchestrate as
part of the package in this diagram.

5. Drag the Load Applicant Data interface into the diagram.

6. Drag the Load Permit Data interface into the diagram.

7. Click the OK connector icon on the package toolbar.

8. Drag the mouse from the Load Applicant Data step to the Load Permit Data step to
create the connection between the 2. This will make Load Permit Data run after Load
Applicant Data completes successfully.

Page 61 of 79
Confidential
9. Click on the Internet folder from the toolbar.

10. Click on the OdiSendMail step and then click on the diagram.

11. Click on the step and rename it to Error Notification from the properties section below
the diagram. Dont worry about the mail server configuration for now, well just make this
error out.

12. Click the KO connector icon on the package toolbar.

13. Drag the mouse from the Load Applicant Data step to the Error Notification step to
create an exception path. This will make the Error Notification step run if the Load
Applicant Data throws an exception.

14. Create the same exception path for Load Permit Data.

Page 62 of 79
Confidential
Archive the Input File

As the last step in the package after successfully loading the permit data, we need to move the
source file into an archive directory with the current timestamp appended to the filename.

1. First, lets create a variable for the current timestamp. From your Integration Project,
right click on Variables and select Insert Variable. For the name, use
CurrentTimestamp

2. Click the Refreshing tab. Select PERMIT_APP_TARGET for the Schema and enter the
following SQL for the Select Query enter:

SELECT TO_CHAR(SYSDATE,'YYYYMMDDHH24MISS') FROM DUAL

Page 63 of 79
Confidential
3. Drag the variable to the package diagram and create an OK connection from Load
Permit Data to CurrentTimestamp.

4. Click on the Files folder from the toolbar.

5. Click the OdiFileMove step and bring it to the diagram and create an OK connection
from CurrentTimestamp.

6. Click on the step to display the properties section below the diagram. Enter the following:

Name: Archive File

Filename: C:\myfiles\CSV\permit_batch.csv
Target File: C:\myfiles\archive\permit_batch_#CurrentTimestamp.csv

Note: #CurrentTimestamp will get the value from the variable at

runtime to append it to the file name.

7. You should see the following:

Page 64 of 79
Confidential
Execute the Package

1. Execute the interface and verify through the ODI Operator it completed successfully.

2. Verify that the archived file exists in C:\myfiles\archive.

Page 65 of 79
Confidential
Lab 7: Setup Agent
Time estimate: 20 minutes

Once ODI Scenarios are created they can be scheduled and orchestrated using an ODI Agent.
The ODI agent is a lightweight java process that orchestrates the execution of ODI Scenarios.
The ODI Agent can be installed as a service to allow for lights out processing of ODI
Scenarios.

Setup ODI Parameters for Agent

ODI Agent will use the repository connection information stored in the file <ODI-
installation>\bin\odiparams.bat. For security, the password information is stored in this file
as an encrypted

1. Generate password to access Master Repository from command line or service

<ODI Installation>\bin\Agent encode master

where master is the password to access work repository in this lab document

When the password is generated, copy the generated encrypted password for later usage

Note that, the ODI_ENCODED_PASS is set to the encoded password for SUNOPSIS by
default; this does not need to be modified unless the SUPERVISOR password has been
changed.

2. Open the odiparams.bat with any text editor, and edit the following parameters:

set ODI_SECU_DRIVER=oracle.jdbc.driver.OracleDriver
set ODI_SECU_URL=jdbc:oracle:thin:@localhost:1521:xe
set ODI_SECU_USER=master
set ODI_SECU_ENCODED_PASS=<your encrypted password>
set ODI_SECU_WORK_REP=WORKREP1
set ODI_USER=SUPERVISOR
set ODI_ENCODED_PASS=LELKIELGLJMDLKMGHEHJDBGBGFDGGH

Note: You can copy values from Designer Connection Dialog, except the encrypted passwords

3. Save this file

With this information, the agent could be executed directly from the command line or
installed as the service to run in the background

Page 66 of 79
Confidential
Create a Physical Agent

1. Go to ODI Topology Designer and click on the Physical Architecture tab.

2. Right click Agents

3. Select Insert Agent and the following screen apprears

Set values as following

Name: Localagent
Host: Localhost
Port: 20910

4. Launch the agent: Start menu > Programs > Oracle > Oracle Data Integrator > Agent

The following screen should apprear

Page 67 of 79
Confidential
5. Click button Test and the following screen should apprear

6. Click Yes and the following screen apprears

7. Click Yes and the following screen apprears

If the test is not successful, please make sure the agent service is started or fix any
OIDPARAMS.bat errors

8. Close the Agent window that you ran in step 4)

The Physical Agent is now setup successfully

Page 68 of 79
Confidential
Create a Logical Agent

1. Click the Logical Architecture tab

2. Right click Agents and click Insert Logical Agent

3. The following screen should apprear

Set the Name: Localagent

Set the Physical Agents: Localagent

Page 69 of 79
Confidential
4. Click OK

The Physical and Logical Agent are now setup successfully. The ODI Agent can now be used
to execute ODI objects

Execute the Agent (use command line)

Make sure that you already have at least 1 scenario to execute

1. Close any Agent windows if you have executed

2. Run the Agent Scheduler in the <ODI-installation>\bin (case sensitive)

Agentscheduler name=Localagent

3. The following screen should appear

4. Check the numbers of scheduled executions is equal to your scheduled scenarios. If you
do not run the lab 8 below, the number should be zero (0).

You have now successfully completed the lab, which walked through the steps that are needed
to setup the ODI Agent. The lab also walked through the steps to execute an ODI object on
the newly setup ODI Agent.

Page 70 of 79
Confidential
Lab 8: Create and Schedule an ODI Scenario
Time estimate: 20 minutes

When a set of objects are complete and tested it is a good practice is to create ODI Scenarios
for each object. Once a scenario is created it cannot be edited and the code for the object is
regenerated and stored in the scenario. Once a scenario is created it can also be scheduled to
be run on virtually any interval within ODI.

Create an ODI Scenario

1. Go to ODI Designer and click on the Projects tab.

2. Expand the Permit Integration folder, right-click on the Packages folder and select the
package Run Permit Integration

3. Click Generate Scenario

4. Name the scenario Run Permit Integration and set the version to 001

5. Expand the scenario

Page 71 of 79
Confidential
6. Right click the scenario to view options

The scenario has now been successfully created, you can now execute the scenario directly,
use the scenario within a package or schedule the package within ODI.

Schedule a New Scenario within Oracle Data Integrator

1. To schedule the package expand the scenario Run Permit Integration

2. Click Insert Scheduling

If an agent is not setup within the ODI Topology Manager, the following message will appear
and the scenario cannot be scheduled.

To schedule a scenario an Agent must be setup. To setup an ODI Agent, see the lab 7 Setup
an ODI Agent

Page 72 of 79
Confidential
3. Click OK to see scheduling options

Once an ODI agent is setup correctly the scenario can be scheduled. Make sure to start the
agent using (case sensitive), you must run only 1 screen

Agentscheduler.bat NAME=Localagent

to start agent as a scheduler

4. Select tab [Definition] and set the values as following

Context : Global
Agent : Localagent
Log Level : 5
Executtion : Simple

Page 73 of 79
Confidential
5. Select tab Execution Cycle to set the execution frequency and contraints when failure

Set Repetition:
Many times: Interval between Repetitions : 1 minutes
Constraints:
Number of Attempts on Failure : 10
Stop Executive After : 5 minutes

6. Click OK. An entry will now be inserted under Scheduling

7. Start the Topology Manager to review scheduling of the agent

8. Right Click and click Edit

Page 74 of 79
Confidential
9. Click Update Scheduling Information. The following screen appears

10. Click OK

11. Click Scheduling Information. The following screen appears

12. Check the execution of the job in the ODI Operator

Page 75 of 79
Confidential
13. Refresh Operator and job should be executed at appropriate time.

You have now successfully scheduled a scenario with an agent scheduler, and the scenario
was executed by the agent successfully.

Congratulations! You have completed the Oracle Data Integrator Hands-On Workshop
for Public Sector. For more learning material on ODI go to:

http://www.oracle.com/technology/products/oracle-data-integrator/index.html

Page 76 of 79
Confidential
Appendix A: Importing the ODI Repository Schemas
If you wish to import the ODI Metadata repository, or if you wish to go back to a fresh copy
of the repository, please follow the instructions below.

Note that running this script will drop the MASTER and WORKREP1 database
schemas.

1. If you have not unzipped the workshop files unzipped to c:\MyFiles, unzip the
ODIWorkshop-MyFiles.zip file from the workshop DVD into your c: drive. This will
create the c:\MyFiles directory containing the workshop files.

Important: Shutdown all the ODI Applications (Topology Manager, Designer and
Operator) before you begin importing.

2. Start a command prompt window. Go to Windows Start > Run and enter cmd.

3. From the command prompt, navigate to c:\MyFiles and enter the following commands:

ImportODIRepositories.cmd

Enter the system password used to install Oracle XE.

This command drops and recreates a new version of the ODI Master and Work Repositories.

Page 77 of 79
Confidential
Appendix B: Resetting the Permit Application Scenario
If you wish to reset the Permit Application target database, the Background Check and the
Permit Request File, please follow the instructions below.

1. Make sure journalization is disabled for the BGCHECK data store.

a. Right-click on the BGCHECK data store and from the Models and select Change
Data Capture > Drop Journal.

a. Right-click on the BGCHECK data store and select Change Data Capture >
Remove from CDC

2. Make sure journalization is disabled for the Load Permit Data interface.

a. Open the Load Permit Data interface from the Projects and go to the diagram.

b. Click on the BGCHECK source node header. From the source configuration
below the diagram, unmark the Journalized Data only option as shown below.

Page 78 of 79
Confidential
3. Start a command prompt window. Go to Windows Start > Run and enter cmd.

4. From the command prompt, navigate to c:\MyFiles and enter the following commands:

resetPermitScenario.cmd

This command truncates the Permit Application target tables, removes the new data inserted
into the Background Check database during the CDC lab, clears the ODI Error tables and
moves the Permit Request File back to its original location.

Page 79 of 79
Confidential