Professional Documents
Culture Documents
2004 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice
The training is designed to have you start using PowerCenter, not to make you experts Youll know how to:
Create logical and physical data flow Design some simple transformations Choose appropriate transformation for your processing Schedule and execute jobs Examine runtime log files Debug transformations and solve data quality issues Run Informatica logic from command line Manage security in Powercenter
2
Agenda Day 1
Introduction Software Installation and Configuration DW/BI Basics Database, Data Warehouse, OLAP/BI, Enterprise Data Warehouse Architecture, ETL & Data Integration, Fitment - Informatica ETL in Data warehouse architecture. Informatica Architecture Brief Description of Informatica Tool & Components (Repository Administrator, Repository Manager, Workflow Manager, Workflow Monitor, Designers, Repository Server, Repository Agent, Informatica Server) LAB: Informatica Installation & Configuration
ETL Components Designer (Source Analyzer, Workflow Designer, Task Designer, Mapplet Designer & Mapping Designer) Workflow Manager (Task Designer, Worklet Designer, Workflow Designer, Workflow Monitor) Repository Manager & Server Informatica Administration Basics LAB: Informatica Administration
Transformations Classifications Active/Passive Re-usable Connected/Un-Connected Transformations and Properties Source Qualifier , Expressions, Lookup, Router, SeqGen, Update Strategy, Targets, Joiner, Filter, Aggregator, Sorter LAB: Transformations - Demo
Agenda Day 5
Mapplets and Mappings Mapping Design Mapping Development Mapping Parameters and variables Incorporating Shortcuts Using Debugger Re-usable transformations & Mapplets Importing Sources & Targets Versioning Overview LAB: Mapplet & Mapping Designing
Agenda Day 6
Development of Mappings Sample mappings LAB: Mapping designing for Flat files loading, DB
Agenda Day 7
LAB: Workflow Designing, Worklet Designing, Task Designing, Scheduling, Sample Workflows, etc.
Agenda Day 8
Advanced Topics Revisit Informatica Architecture Server Configurations (pmServer/pmRepServer), Memory Management Caches (Lookups, Aggregators, Types of Caches) Performance Tuning Release Management & Versioning Repository Metadata Overview LAB: Performance Tuning Demo, Release
Agenda Day 9
GMAS ETL Process & Development Methodology Design/Development Guidelines, Checklists, etc. Best Practices & references my.Informatica.com, Informatica discussion groups etc Question & Clarification sessions LAB Sessions Sample mappings/workflows
10
Informatica PowerCenter
PowerCenter is an ETL tool: Extract, Transform, Load A number of Connectivity options (DB-Specific, ODBC, Flat, XML, other) Metadata Repository/Versioning built in Integrated scheduler (possibility to use external) Number of cool features XML Imports/Exports, integrity reports, Save As Picture etc
12
PowerCenter Concepts
Repository: stores all information about definition of processes and execution flow Repository server: provides information from the Repository Server: executes operations on the data
Must have access to sources/targets Has memory allocated for cache, processing
Clients (Designer, workflow manager etc): manage the Repository Sources, Targets can be over any network (local, FTP, ODBC, other)
13
PowerCenter Concepts II
Client s Rep Server Server(s )
Sources
Repositor y
Targets
14
Software installation
15
You need to tell your client tools where is the Repository Sever Go to the Repository Manager, Choose Repository->Add Repository . Repository is <Server Name>, user is your Name
16
Right-click on the newly created repository, choose Connect Fill in additional details. Password, port, etc.
17
Add the connection information to tnsnames.ora on your machine Verify the connection using SQLPLUS
18
This connection will be used to connect from Client tools directly to a database (e.g. for imports) Go to Control Panel -> Administrative Tools -> Data Sources (ODBC) On the System DSN tab click Add and choose Microsoft ODBC for Oracle. Fill in details for your TNumberOL
19
20
STOP
Wait for the class to finish the connectivity setup
Sources, Targets
Designer defines Sources, Targets, Transformations and their groups (Mapplets /Mappings) Workflow Manager defines Tasks and Workflows (+scheduling) Workflow Monitor monitors job execution Repository Manager defines connection to repository Server Administration Console for Admins only. Must have English locale settings on PC
23
Transformation
selector
Tool Selector
Navigator Workspace
Messages
24
The data flow structure is defined in the Designer client tool Following concepts are used:
Sources: structure of the source data Targets: structure of the destination data Transformation: an operation on the data Mapplet: reusable combination of Transformations Mapping: complete data flow from Source to Target
25
Target(s)
Source(s)
Transformation(s)
26
Sources - overview
Sources define structure of the source data (not where the data is). Source + Connection = complete information It is only the internal Informatica information, not e.g. physical structure in the database. Sources are created using the Source Analyzer in the Designer client tool You can either create or import Source definitions
27
Go to Tools -> Source Analyzer Choose Import from the database Choose the waretl ODBC connection and fill in remaining details
If you get unable to resolve TNS name error make sure you have added the waretl server to all tnsnames.ora files
28
29
30
To Edit a Source doubleclick on it in the Workspace You can manually add/delete/edit columns You can define PK/FK/other relations in INFA You can load based on PK (Unit). See the Update Strategy Transformation
31
Sources: Exercise 1
Define this comma-delimited source in Source Analyzer. Use name SRC_TNumber_Flat Preview data in your file in the Source Analyzer.
32
STOP
Wait for the rest of the class to finish If you finish early, read the Designer Guide -> Chapters 2 and 3
Targets overview
Targets define the structure of the destination objects Target + Connection = complete information It is only the internal Informatica information, not e.g. physical structure in the database. Targets are created using the Warehouse Designer in the Designer client tool You can either create or import Target definitions Defining Targets is works the same way as defining Sources
34
Targets - Columns
35
Targets: Exercise I
Import table TGT_TNumber_1 to Warehouse Designer using Import from Database function Compare your Target with the Flat File Source SRC_TNumber_Flat Modify the your Target to be able to load all data from your Flat File. Remember to have your Oracle and INFA definitions synchronized!
This can be done literally <1 minute :)
36
Targets: Exercise II
Define all columns from our Flat File source plus new ones:
FACT_ID (PK) GEO_ID
Create the Target in Oracle too! (and dont forget grants to dev_rol1..)
This too can be done <1 minute :>
37
STOP
Wait for the rest of the class to finish If you finish early, read the Designer Guide -> Chapter 4
Transformations, Mappings
Transformations - Overview
Transformations modify the data during processing. They are internal INFA objects only Transformations can be reusable Large number of transformations available
E.g. Lookup, Filter, Aggregator.. Every transformation has different configuration options
Transformations Overview II
Very good online help available Transformation Guide. Use it for self-study!
41
Transformations - Concepts
42
Transformations Concepts II
43
Mappings Overview
Mapping is a definition of an actual end-to-end processing (from Source to Target) You connect Transformations by Ports defining a data flow (SH30->CHAIN_TYPE/CHAIN) The data flows internally in INFA on the Server You can visualize the data flowing row by row from a Source to a Target
Exceptions: Sorters, Aggregators etc
Mappings - Creating
45
Begins the process flow from physical Source onwards Can select only a couple of columns from Source (will SELECT only those!) Can Join multiple sources (PK/FK relationship inside INFA, not Oracle) For relational sources:
You can override the where condition Can Sort the input data Can Filter the input data
46
Transformations: SQ II
For relational sources you can completely Override the SQL statement Is created by default when you drag a Source to the Mapping Designer Standard naming convention is SQ Some options are available only for Relational sources (e.g. sorting, distinct etc) As usually more info in the Transformation Guide
Self-study on overwriting the default SQL ad Where conditions
47
SQ: Creating
Having Mapping Designer open, drag your Flat File SRC_TNumber_Flat onto Workspace SQ is created automatically for you SQ is often a non-reusable component
48
49
Workspace
50
Exercise: SQ
Create manually an SQ for your Source, the ports and links should be connected automatically
Hint: theres a Transformations menu available when in Mapping designer Hint: you can drag Ports to create them in the target Transformation Save your work the Mapping is Invalid. Why?
51
Having Mapping Designer open, drag the TGT_TNumber_1 Target onto Workspace Connect appropriate Ports from SQ to Target
Save - the Mapping is now Valid! :) Our Mapping is actually a SQL-Loader equivalent
52
STOP
Wait for the rest of the class to finish
Connections, Sessions
Execution of a Mapping
A single Mapping can be executed over different connections An executable instance of a Mapping is called a Session A series of Sessions is a Workflow Workflows are defined in Workflow Designer
55
Workflow Designer
56
Workflows
The parameters are used to define physical locations for files, logs etc They are relative to Server, not your local PC! Directories must be accessible from Server
57
Connections
Connections define WHERE to connect for a number of reasons (Sources, Targets, Lookups..) There are many Connections types available (any relational, local/flat file, FTP..) Connections are defined in the Workflow Manager Connections have their own permissions! (owner/group/others)
58
Connections: Exercise
Choose New Connection of type Oracle and fill in the necessary info
59
Tasks
Tasks have huge number of attributes that define WHERE and HOW the task is executed Remember, online manual is your friend :)
60
Sessions : Creating
Set the Workspace to Task Developer: Go to Tools->Task Developer Go to Tasks->Create, choose Session as a Task Type Create a task called tskTNumber_Load_1
61
Sessions : Creating II
62
Properties tab:
Session log file directory (default $PMSessionLogDir\) and file name $Source and $Target variables: for Sources/Targets, Lookups etc. Use Connections here (also variables :) ) Treat Source Rows As: defines how the data is loaded.
Read about the Update Strategy Transformation to understand how to use this property. This is very useful property, e.g. able to substitute concept of a Unit
63
Multiple error handling options Error Log File/Directory ($PMBadFileDir) All options in the Config Object tab are based on the Session Configuration (a set of predefined options). To predefine go to Tasks>Session Configuration
64
Mapping tab defines the WHERE Connections, Logs For every Source and Target define type of connection (File Writer, Relational..) and details
65
For relational Sources and Targets you can/must define owner (schema) Click on the Source/Target on the Connection tab The Attribute is :
Owner Name for Sources Table Name Prefix for Targets
If you use a shared Connection and access private objects (without public synonyms) you MUST populate this attribute
66
Components tab
Pre, post session commands Email settings
67
Sessions : Summary
Lot of options again, read the online manual Most importantly, you define all Wheres: $Source, $Target, Connections for all Sources, Targets, Lookups etc Heres also definition of locations of flat files Define error handling/log locations for every Task Use Session Configs Majority of the Session options can be overwritten in Workflows :)
This allows e.g. to execute the same Session over different Connections!
68
Sessions: Exercise
STOP
Wait for the class to finish
Workflow execution
Workflows: Creating
Choose Workflows->Create
Create a Workflow wrkTNumber_Load_1
72
Workflows: Properties
73
Workflows: Properties II
Properties tab:
Parameter filename: holds list of Parameters used in Mappings. See online manual Workflow log (different than Session logs!)
74
Scheduler tab:
Allows to define complete reusable calendars Explore on your own! First read the online manual, then schedule some jobs and see what happens :)
75
Workflows: Properties IV
Variables tab
Variables are also used during Mapping execution Quick task: Find what is the difference between a Parameter and a Variable
Events tab
Add user-defined Events here These events are used later on to Raise or Wait for a signal (Event)
76
With your Workflow open drag the tskTNumber_Load_1 Session onto the Workspace Go to Tasks and choose Link Tasks Link the Start Task with tskTNumber_Load_1
The Start Task does not have any interesting properties
77
You can edit Task properties in a Workflow the same way as you do it for a single Task
Editing the task properties in Workflow overwrites the default Task properties Overwrite only to change the default Task behavior Use system variables if a Task will be executed e.g. every time on a different instance
78
Workflows: Running
Before you run a Workflow, run the Workflow Monitor and connect to Server first!
79
Workflows: Running II
In Workflow Designer right-click on the Workflow wrkTNumber_Load_1 and choose Start Workflow Go to Workflow Monitor to monitor your Workflow
80
Workflows: Running II
Workflow Monitor displays Workflow status by Repository/Server/Workflow/Session Two views available: GANTT view and TASK view
81
Workflows: Logs
You can get the Workflow/Session log by rightclicking on Workflow/Session and choosing the log
Remember, Session log is different than Workflow log!
82
Workflows: Logs II
Most interesting information is in the Session logs (e.g. Oracle errors etc) Exercise:
Where do you define location of Where are the Session/Workflow logs? Manually locate and open the logs for your Workflow run Find out why your Workflow has failed :>
83
84
Workflows: Restarting
Restart your Workflow using Restart Workflow from Task (More about job restarting: Workflow Administration Guide -> Chapter 14 ->Working with Tasks and Workflows ) Debug until your Workflow finishes OK
85
Workflow: verifying
Check that the data is in your Target table. The table will be empty initially why? Why theres an Oracle error in the Session Log about Truncating the Target table? Hint: when you modify a Mapping that is already used in a Session, you need to refresh and save the Session. Warning! PowerCenter has problems refreshing objects between tools! Use Task->Edit or File>Close All Tools
86
Create a new Mapping m_TNumberLoad_2 that will move all data from TGT_TNumber_1 to TGT_TNumber_2 Create a new session tskTNumber_Load_2 for this Mapping. Define connection information to waretl, check Truncate target option and use Normal load mode (NOT Bulk) Create a new Workflow wrkTNumber_Load_2 that will run tskTNumber_Load_2 Run your Workflow, make sure it finishes OK Check that the data is in TGT_TNumber_2 table.
87
STOP
Wait for the class to finish
Define structures of Sources and Targets Define where the Sources and Targets are Create a simple Workflow Run a Workflow Debug the Workflow when it fails
Transformations
Expressions, Sequences
Well be going through a number of Transformations Only some (important) properties will be mentioned Read the Transformation Guide to learn more
91
92
Expression Editor
93
EXP: Example
Lets create an non-reusable Expression that will change customer Bolek into customer Lolek You can drag ports to copy them IIF function similar to DECODE
94
95
SEQ: Ports
Nextval
96
SEQ: Properties
Start Value Increment By The difference between two consecutive values from the NEXTVAL port. End Value: The maximum value the PowerCenter Server generates. Current Value: The current value of the sequence. Cycle Number of Cached Values: The number of sequential values cached at a time. Use when multiple sessions use the same reusable SEQ at the same time to ensure each session receives unique values. 0 = caching disabled Reset: Rewind to Start Value every time a Session runs Disabled for reusable SEQs
97
98
Transformations: Exercise
Create a Target called TGT_TNumber_Tmp that will hold FACT_ID, SALES and SALESx2 columns Create appropriate table in Oracle, (remember: grants, synonyms if needed..) Add this Target to your Mapping m_TNumberLoad_2 (so, you should have two Targets). Save. The Mapping is valid even though the Target is not connected why? In the Workflow define Connection for this Target (remember, use Normal load mode). Choose Truncate Target option
99
EXP: Exercise
Create a reusable expression called EXP_X2 that will multiply an integer number by two. The input number must be accessible after the transformation Use this EXP to multiply the SALES field when passing it to TGT_TNumber_Tmp. The original SALES field goes to SALES and the multiplied one to SALESx2
100
SEQ: Exercise
Create a reusable SEQ called SEQ_TNumber_FACT_ID. Start = 1, Increment By = 6, Cache = 2000 Populate the FACT_ID field in both targets to have the same value from the sequence (parent/detail).
101
Exercise: verify
You may need to modify the Workflow before you run it - what information needs to be added? Verify that the data is in both TGT_TNumber_2 and TGT_TNumber_Tmp, the SALES_x2 column equals SALES*2 and the same FACT_ID is used for the same row in _2 and _TMP table. Rerun the workflow a couple of times. What happens with FACT_ID field on every run?
102
Solutions:
Wrong! SEQ initialized for every target. _TMP and _2 tables will have different FACT_IDs
103
Solutions:
Wrong! SEQ initialized for every target. _TMP and _2 tables will have different FACT_IDs
104
105
STOP
Wait for the class to finish
The Debugger
The Debugger
Debugger allows you to see every row that passes through a Mapping
Every transformation can be debugged in your Mapping Nice feature available: breakpoint The Debugger is available from the Mapping Designer
To start the debugger, open your Mapping in Mapping Designer and go to Mappings -> Debugger -> Start Debugger Well be debugging the m_TNumberLoad_2 Mapping Warning: the Debugger uses a lot of server resources!
108
109
All properties of this session (Connections etc) will be used in the debug session
110
Select the Targets you want to debug. This defines flow in the Debugger You have an option to discard the loaded data
111
Flow Monitor
Target data
112
Select EXP_x2 as the Transformation youll be monitoring Remember, you run the Debug session for one Target! Most optimal operation with the Debuger:
Choose your mapping on the Workspace Choose Debugger -> Step to Instance. Goes directly to your Transformation Or, Choose Debugger -> Next instance. This is actually step-by!
113
You can view individual Transformations in the Instance window. The data is shown with regard to given data row:
Continue = run the Session (if there are no Breakpoints the Session with run until OK/Failure When the whole Source is read, the Debugger finishes
114
You can add breakpoints to have the Debugger stop on given condition
Mappings -> Debugger -> Edit Breakpoints
115
Breakpoints: Setting up
116
You can actually change the data in Debugger flow! Check out this feature, really useful.
A number of restrictions apply: usually can modify only output ports and group conditions
117
STOP
Wait for the class to finish
Transformations
Lookups
120
Connected lookups receive data from the flow while Unconnected are inline functions Unconnected lookup has one port of type R = Return
Connected Default values Caching #returned columns Supported Any type Multiple Unconnected Not supported (possible using NVL) Static only Only one
121
Lookups: Creating
While in the Transformation Designer choose Transformations -> Create and create lkpRefCtrl
122
Lookups: Creating II
123
124
Input ports define comparison columns in the INFA data flow Lookup ports are columns in the Lookup table For unconnected Lookup there one R port the return column
125
126
Match the Iinput ports with Lookup ports on the Condition tab
127
Lookup Sql Override: you can actually override the Join. More on Overriding in the Designer Guide Lookup table name: Join table Lookup caching enabled: disable for our Exercise Lookup policy on multiple match: first or last Connection Information: very important! Use $Source, $Target or other Connection information
128
129
Connected Lookups dont have Return port (they use Output ports) Data for the lookup comes from the pipeline
130
Lookups: Exercise
Modify Mapping m_TNumberLoad_2 to find a ISO_CNTRY_NUM for every customer. Column GEO_ID should be populated with ISO_CNTRY_NUM. Use default values if CUST_ID not found Choose Connected or Unconnected Lookup Run your modified Workflow
Verify that when your Workflow finishes OK all rows have GEO_ID populated
131
STOP
Wait for the class to finish
Transformations
Stored Procedures
Used to execute an external (database) procedure Naming convention is name of the Procedure Huge number of different configurations possible
Connected, Unconnected (same as Lookups) Pre- and post-session/load Returned parameters
Well do an example of a simple stored procedure Possible applications: e.g. Oracle Partitioning for promotions in Informatica (post-session)
PowerCenter has hard time running inlines!
134
The easiest way is to Import the procedure from DB: Transformation -> Import SP
135
136
137
Stored Procedure
138
STOP
Wait for the class to finish
Transformations Overview
Aggregator, Filter, Router, Joiner, Update Strategy, Transaction Control, Sorter, Variables
Overviews
Number of interesting Transformations and techniques outside of the scope of this training Overview gives you an idea that a possibility exists to do something If you want to learn more self study: read the Designer Guide and the Transformation Guide
141
142
Allows to reject rows that dont meet specified criteria. Rows filtered are not in the reject file.
Active transformation
143
Typical usage:
144
Router: configuring
145
Router receives the whole stream and sends it different way depending on conditions
146
Restrictions:
Cant use is either input pipeline contains an Update Strategy transformation Cant use if one connects a Sequence Generator transformation directly before the Joiner transformation Allows to join two pipelines. If more joins needed, use consecutive JNRs
147
Joiner: Example
148
Joiner: Ports
One pipeline is master, the other one is detail. M port denotes which one is which
149
Joiner: Properties
Properties tab lets you define join type (amongst other properties)
150
This transformation lets you mark a row as Update, Insert, Delete or Reject You do it by a conditional expression in the Properties tab of UPD
E.g. :IIF( ( SALES_DATE > TODAY), DD_REJECT, DD_UPDATE )
151
Then enter the conditional expression into Properties tab of UPD. Use variables:
Insert DD_INSERT 0 Update DD_UPDATE 1 Delete DD_DELETE 2 Reject DD_REJECT 3
You must set properly Session properties. Read more in the Transformation Guide
For example you must select Treat Source Rows As Session Property option to Data Driven
152
This transformation defines commit points in the pipeline To setup, populate the conditional clause in Properties
For example IIF(value = 1, TC_COMMIT_BEFORE, TC_CONTINUE_TRANSACTION)
153
Transaction control must be effective for every source, otherwise the mapping is invalid
Read more in the Transformation Guide
154
155
Variable ports
156
Useful when e.g. running inlines for distinct values You need to create two variable Ports to store values from previous rows (why? :> )
157
Transformations: Exercise
Modify the m_TNumberLoad_2 mapping to use JNR transformation for geography lookup (instead of LKP) Count the number of distinct customers in the pipeline (modify the Target table to have CNT column). Use Variables
Once you finish, modify the Transformation to complete the task without Variables
Change back your mapping to use Lookup instead of Joiner
158
Transformations: Exercise II
Build a Loader mapping just with the following objects: Source, SQ, UPD, Target Add a PK on your Source (e.g. TRANX_ID) Add TRANX_ID to your flat file Use the UPD strategy to insert new rows and update already existing rows (based on TRANX_ID field)
Remember, set correct Session parameters
Load your flat file Verify: add and modify some rows in the flat file. Load again, check that the Target is updated as needed (rows are added/modified/deleted)
159
STOP
Wait for the class to finish
Mapplets, Worklets
Mapplets: Overview
Mapplets are reusable sets of Transformations put together into a logical unit Can contain Sources and some Transformations. Cant contain Targets or other Mapplets
162
Mapplets in Mapping
163
Worklets: Overview
164
Mapplets: Exercise
Create a table called EMP_DEPT which has EMPNO,MGR, DEPT_NAME as fields (Take the structure of the fields from EMP table) Create a mapping map_EMP_NAME which has EMP1 as Source and EMP_DEPT as target and use the above mapplet inside this mapping. Run the Mapping, verify results
165
STOP
Wait for the class to finish
Advanced Scheduling
Advanced Scheduling
When you build Workflows in the Workflow Designer you can use a number of non-reusable components Different control techniques available To use the non-reusable components go to Tasks -> Create when in Workflow Designer
168
Workflows: Links
Task properties:
169
Expression Editor
Links (and some Tasks) can use the Expression Editor for Workflow events
170
Tasks: Command
Command Task executes any script on the Informatica Server It can be reusable Property Run if previous completed controls execution flow when more than one script is defined
171
Tasks: Decision
172
Workflow: Variables
173
Workflow: Variables II
174
Tasks: Assignment
175
Tasks: Email
Emails: Advanced
177
Additional options are available for Email body when using from within a Session
178
Events: Overview
Events are a way of sending a signal from one Task to another Two types of Events supported:
User-defined (define in Workfow->Events) Filewatcher event
179
Events: User-defined
180
Events: Example
181
182
Properties available
Enable Past Events!
183
The filewatcher event is designed to wait for a marker file (e.g. *.end)
Theres an option to delete the file immediately after the filewatcher kicks in No wildcards are allowed
Discussion: how to emulate S1 filewatcher, waiting and loading for multiple files?
184
Tasks: Control
185
Tasks: Timer
186
Tasks: Exercise
Create a new Workflow that will use two Sessions: ..Load_1 and ..Load_2 Run the ..Load_2 Session after every 3 runs of ..Load_1
Dont cycle the Sessions, rerun the whole Workflow How will you verify success?
Obligatory tasks:
Decision Event Raise/Wait
187
STOP
Wait for the class to finish
pmcmd - overview
A command line tool to execute commands directly on the Informatica server Useful e.g. to:
Schedule Powercenter tasks using an external scheduler Get status of the server and its tasks
190
pmcmd - example
pmcmd getserverdetails <-serveraddr|-s> [host:]portno <<-user|-u> username|<-uservar|-uv> userEnvVar> <<-password|-p> password|<-passwordvar|-pv> passwordEnvVar> [-all|-running|-scheduled]
191
pmrep - overview
Very useful for e.g. bulk operations (create 10 users) Usage: pmrep command_name [-option1] argument_1 [-option2] argument_2...
192
pmrep - usage
The last command must be exit Full list of commands in the Repository Guide -> Chapter 16: using pmrep
193
pmrep exit
pmcmd: exercise
Run the workflow wrkTNumber_Load_2 from command line: pmcmd startworkflow <-serveraddr|-s> [host:]portno <<-user|-u> username|<-uservar|-uv> userEnvVar> <<-password|-p> password|<-passwordvar|-pv> passwordEnvVar> [<-folder|-f> folder] [<-startfrom> taskInstancePath] [-recovery] [-paramfile paramfile] [<-localparamfile|-lpf> localparamfile] [-wait|-nowait] workflow
194
STOP
Wait for the class to finish
Parameters dont change but Variables change between Session runs. The changes are persistent Both Variables and Parameters can be defined in the Parameter File
Except port variables Variables can initialize without being defined upfront
197
Described in the Designer Guide -> Chapter 8 Use them inside transformations in regular expressions
E.g. in SQs (WHERE), Filters, Routers
198
199
Session Parameters
Very useful! Can be used to have the same Session work on different files/connections Must be defined in the Parameter File Conventions:
Parameter Type Database Connection Source File Target File Lookup File Naming Convention $DBConnectionName $InputFileName $OutputFileName $LookupFileName
Reject File
$BadFileName
200
You can replace majority of the Session attributes with Parameters Described in detail in the Workflow Administration Guide -> Chapter 18: Session Parameters
201
Parameter File
The variable values in the file take precedence over the values saved in the Repository
This means that if a Variable is defined in a Parameter File, the change of its value in Mapping will have no effect when the Session runs again!
Described in detail in the Workflow Administration Guide -> Chapter 19: Parameter Files
202
Parameter Files II
Parameter Files can be put on the Informatica Server machine or on a local machine
Local files only for pmcmd use
A single parameter file can have sections to hold ALL parameters and variables
203
204
Modify the map_EMP_NAME Mapping to load only one Employee specified by a Parameter
Remember to define the Parameter in the Parameter File
Modify the Mapping to store the SAL as SAL+(SAL*30/100) (increase salary by 30%) Use Mapping Variables Test!
206
Modify the S_EMP_NAME Mapping to use a Parameter for the file name to be loaded
Remember to define the Parameter in the Parameter File
How would you load e.g. 10 files one after another using the same Session?
207
STOP
Wait for the class to finish
Security overview
Security in PowerCenter
Security topics are described in Repository Guide -> Chapter 5; Repository Security PowerCenter manages privileges internally
Repository privileges (individual or group) Folder permissions Connection privileges
Users, Groups
Individual (User) and Group privileges are combined to get the overall view on someones permissions The group Administrators has all possible privs
211
Repository privileges
Repository privileges are granted to Groups and Users The Repository privileges work on Objects! Detailed description of Repository privileges is in the Repository Guide -> Chapter 5 -> Repository Privileges
212
Object permissions
213
Performance tuning
Good overview in the Workflow Administration Guide -> Chapter 25: Performance Tuning
215
Bottlenecks: Identifying
Target bottlenecks
If Target is relational or remote location, change to local Flat file and compare run time
LAN speed can affect the performance dramatically for remote Sources/Targets
Query remotely/locally to identify LAN problems
217
Mapping bottlenecks
Put Filters just before Targets: if the run time about the same you may have a Mapping bottleneck Some transformations are obvious candidates
Lookups
Multiple Transformation Errors slow down transformations Use Performance Details file
218
The Performance Detail File has very useful information about every single transformation
File is created in the SessionLog directory Big number of performance statistics available Workflow Administration Guide -> Chapter 14 Monitoring Workflows -> Creating and Viewing Performance Details
219
220
Usually related to insufficient cache or buffer sizes Use the Performance File
Any value other than zero in the readfromdisk and writetodisk counters for Aggregator, Joiner, or Rank transformations indicate a session bottleneck.
221
To configure these settings, first determine the number of memory blocks the PowerCenter Server requires to initialize the session. Then, based on default settings, you can calculate the buffer size and/or the buffer block size to create the required number of session blocks.
222
2. Next, based on default settings, you determine that you can change the DTM Buffer Size to 15,000,000, or you can change the Default Buffer Block Size to 54,000:
(session Buffer Blocks) = (.9) * (DTM Buffer Size) / (Default Buffer Block Size) * (number of partitions) 200 = .9 * 14222222 / 64000 * 1 or 200 = .9 * 12000000 / 54000 * 1
223
224
A balanced Session
The Session Log has statistics on Reader/Transformation/Writer threads (at the end of the file) MASTER> PETL_24018 Thread [READER_1_1_1] created Total Run Time = [595.053326] secs, Total Idle Time = [319.658512] secs, Busy Percentage = [46.280695]. MASTER> PETL_24019 Thread [TRANSF_1_1_1] created Total Run Time = [592.979465] secs, Total Idle Time = [248.725231] secs, Busy Percentage = [58.055001]. MASTER> PETL_24022 Thread [WRITER_1_*_1] created Total Run Time = [535.331108] secs, Total Idle
225
Increasing performance
226
Tuning Sources/Targets
Limit SQs
Limit incoming data (# rows) Tune SQ queries Prepare the data on the source side (if possible)
For Targets use Bulk Loading and avoid PKs/Indexes Increase LAN speed for remote connections
227
Tuning Transformations
Use Filters as early in the pipeline as possible Use port variables for complex calculations (factor out common logic) Use single-pass reading
228
Optimizing Sessions/System
229
Pipeline Partitioning is a way to split a single pipeline into multiple processing threads
231
Pipeline Partitioning
This requires multi-CPU machines and relational databases with parallel options enables HUGE performance benefits can be achieved
If you know what youre doing, otherwise you may actually lower system performance!
232
Pipeline partitions are added in the Mapping tab of Session properties (Workflow Manager)
233
Partitioning Limitations
Cant add partition points to these transformations because not all the columns flow through this part of the pipeline
234
Partition Types
Round-robin. The PowerCenter Server distributes data evenly among all partitions. Use round-robin partitioning where you want each partition to process approximately the same number of rows. Hash. The PowerCenter Server applies a hash function to a partition key to group data among partitions. If you select hash auto-keys, the PowerCenter Server uses all grouped or sorted ports as the partition key. If you select hash user keys, you specify a number of ports to form the partition key. Use hash partitioning where you want to ensure that the PowerCenter Server processes groups of rows with the same partition key in the same partition. Key range. You specify one or more ports to form a compound partition key. The PowerCenter Server passes data to each partition depending on the ranges you specify for each port. Use key range partitioning where the sources or targets in the pipeline are partitioned by key range. Pass-through. The PowerCenter Server passes all rows at one partition point to the next partition point without redistributing them. Choose passthrough partitioning where you want to create an additional pipeline stage to improve performance, but do not want to change the distribution of data across partitions. Database partitioning. The PowerCenter Server queries the IBM DB2 system for table partition information and loads partitioned data to the corresponding nodes in the target database. Use database partitioning with IBM DB2 targets stored on a multi-node tablespace. For more information, refer Workflow Administration Guide
235
Migration strategies
Migration Strategies
Usual problems with object synchronization There are two main types of migration
Repository per Stage Folder per stage
237
Folder Migrations
Lower server requirements (one repository) Easier security management (one user login) Folders are created and managed in the Repository Manager
You need to have appropriate privs
238
Repository migrations
In this case you have a separate Repository (not necessarily Repository Server) per Stage Reduces the Repository size/complexity Streamlines folder structure
Test repository
Prod repository
239
Copy Wizard
240
Copy Wizard II
You can copy between repositories or within the same repository The Wizard helps you to resolve conflicts
Connections Variables Folder names Other
241
XML Exports/Imports
If not possible to copy between folders or repositories (e.g. no access at all for Dev group to QA repository), one can use XML Export/Import
242
XML Imports/Exports II
When Exporting/Importing there are all dependencies exported (e.g. Sources for Mappings)
When Importing an Import Wizard will help you to resolve any possible conflicts
Different folder names Existing objects other
243
244
Deployment Groups
For versioned Repositories you can group objects into Deployment Groups Greater flexibility and reduced migration effort You can define whole application or just a part of it No need to have one folder per application A deployment Group can be Static or Dynamic
246
This test checks some skills you should have learned during the course Its supposed to prove your knowledge, not your colleagues or mine Its close to real life development work The test requires from you
Application of gained knowledge use training materials and online guides! Creativity Persistance
248
Your task is to load some data into target database, manipulating it on the way
So, its a typical ETL process
249
Define a workflow that will load the data from file agg_src_file_1.txt to Oracle
Create your own target Oracle table for called ODS_TNUMBER If a numerical value is not numerical then load the row anyway, using 0 for numerical value Use an Update Strategy transformation based on the original transaction ID (ORIG_TRX_ID) to insert new rows and update existing rows Verify:
#rows in = #rows out Sum(values) in the source file = sum(vaules) in the target
250
Move all the data from table ODS_TNUMBER to table QDF_TNUMBER, adding following columns on the way:
TRADE_CHANL_ID from CUST table GEO_NAME from GEO.NAME table, linking via CUST.ISO_CNTRY_NUM
Filter out all rows with sales values <=0 Create your QDF table
251
Create a report (Oracle table) that will give information how much was daily sales in each Sector
Sector is level 2 in the 710 hierarchy Use DNORM table to get SECTOR information Use most recent CTRL_PERD Create appropriate Oracle report table
252
Test rules
Dont change any Oracle or source data, however you may create your own objects
253
Task hints
Some values may be a bit different than other try to fix as many data issues as possible Remember about performance! Large lookups, aggregations, joins Use log files and the Debugger Use reusable components if feasible
254
The End
2004 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice