Professional Documents
Culture Documents
By Ram Reddy
Course Objectives
By the end of this course you will: Understand how to use the major PowerCenter components for development Be able to build basic ETL mappings and mapplets* Be able to create, run and monitor workflows Understand available options for loading target data Be able to troubleshoot most problems
About Informatica
Founded in 1993 Leader in enterprise solution products Headquarters in Redwood City, CA Public company since April 1999 (INFA) 2000+ customers, including over 80% of Fortune 100 Strategic partnerships with Oracle ,IBM, HP, Accenture, SAP, and many others Worldwide distributorship
Informatica Products
PowerCenter PowerAnalyzer ETL batch and real -time data integration BI reporting web-browser interface with reports, dashboards, indicators, alerts; handles real -time metrics Centralized metadata browsing cross-enterprise, cross enterprise, including PowerCenter, PowerAnalyzer, DBMS, BI tools, and data modeling tools Data access to mainframe, mid-size system and complex files
SuperGlue*
PowerExchange
Data access to transactional applications and PowerCenter Connect products real-time services
* Uses PowerCenter to extract metadata and PowerAnalyzer to display reports
#209 Nilagiri Block Adithya Enclave Ameerpet,HYD @8801408841, 8790998182 , , ,
Informatica Resources
www.informatica.com provides information (under Services) on: Professional Services Education Services my.informatica.com sign up to access: Technical Support Product documentation (under Tools online documentation) Velocity Methodology (under Services) Knowledgebase Webzine Mapping templates devnet.informatica.com sign up for Informatica Developers Network Discussion forums Web seminars Technical papers
#209 Nilagiri Block Adithya Enclave Ameerpet,HYD @8801408841, 8790998182 , , ,
Decision Support
Data Warehouse
Transaction level data Optimized for transaction response time Current Normalized or De-normalized data
Aggregate data Cleanse data Consolidate data Apply business rules De-normalize data
Extract
Import from: - Relational database - Flat file - XML object Create manually
DEF
Flat File
DEF
DEF
Data Previewer
Preview data in
Relational database sources Flat file sources Relational database targets Flat file targets
Mappings
3. Mappings
By the end of this section you will be familiar with: The Mapping Designer interface Transformation objects and views Source Qualifier transformation The Expression transformation Mapping validation
Mapping Designer
Transformation Toolbar Mapping List
Iconized Mapping
Transformation Views
A transformation has three views:
Iconized shows the transformation in relation to the rest of the mapping Normal shows the flow of data through the transformation Edit shows transformation ports (= table columns) and properties; allows editing
#209 Nilagiri Block Adithya Enclave Ameerpet,HYD @8801408841, 8790998182 , , ,
Ports
All input/output Convert datatypes For relational sources: Modify SQL statement User Defined Join Source Filter Sorted ports Select DISTINCT Pre/Post SQL
Usage
Character Functions Used to manipulate character data CHRCODE returns the numeric value (ASCII or Unicode) of the first character of the string passed to this function CONCAT is for backward compatibility only. Use || instead
Conversion Functions
Used to process data during data cleansing METAPHONE and SOUNDEX create indexes based on English pronunciation (2 different standards)
Date Functions
Used to round, truncate, or compare dates; extract one part of a date; or perform arithmetic on a date To pass a string to a date function, first use the TO_DATE function to convert it to an date/time datatype
Test Functions Used to test if a lookup result is null Used to validate data
Selected port
Mapping Validation
Connection Validation
Examples of invalid connections in a Mapping:
Connecting ports with incompatible datatypes Connecting output ports to a Source Connecting a Source to anything Except a Source Qualifier or Normalizer transformation* Connecting an output port to an output port or an input port to another input port
Mapping Validation
Mappings must: Be valid for a Session to run Be end-to-end complete and contain valid expressions Pass all data flow rules Mappings are always validated when saved; can be validated without being saved Output Window displays reason for invalidity
Example Mapping
4. Session Task
Server instructions to run the logic of ONE specific mapping
e.g. source and target data location specifications, memory allocation, optional Mapping overrides, scheduling, processing and load instructions
Becomes a component of a Workflow (or Worklet) If configured in the Task Developer, the Session Task is reusable (optional)
5. Workflow Structure
A Workflow is set of instructions for the Informatica Server to perform data transformation and load Combines the logic of Session Tasks, other types of Tasks and Worklets The simplest Workflow is composed of a Start Task, a Link and one other Task
Link
Start Task
Session Task
6. Workflow Monitor
The Workflow Monitor is the tool for monitoring Workflows and Tasks Choose between two views:
Task view
Constraint-based Loading
PK1
FK1 PK2
FK2
To maintain referential integrity, primary keys must be loaded before their corresponding foreign keys here in the order Target1, Target2, Target 3
# 209 Nilagiri Block Adithya Enclave Ameerpet,HYD @8801408841,8790998182 , , ,
Active source
Active transformation that generates rows Cannot match an output row with a distinct input row Examples: Source Qualifier, Aggregator, Joiner, Sorter (The Filter is NOT an active source)
Active group
Group of targets in a mapping being fed by the same active source
Example 1
With only one Active source, rows for Targets1, 2, and 3 will be loaded properly and maintain referential integrity
PK1
Example 2
With two Active sources, it is not possible to control whether rows for Target3 will be loaded before or after those for Target2
Example
Target Tables CREATE TABLE DEPT1 (DEPTNO NUMBER(2) CONSTRAINT PK_DEPT1 PRIMARY KEY, DNAME VARCHAR2(14) , LOC VARCHAR2(13) ) ; CREATE TABLE EMP1 (EMPNO NUMBER(4) CONSTRAINT PK_EMP1 PRIMARY KEY, ENAME VARCHAR2(10), JOB VARCHAR2(9), MGR NUMBER(4), HIREDATE DATE, SAL NUMBER(7,2), COMM NUMBER(7,2), DEPTNO NUMBER(2) CONSTRAINT FK_DEPTNO1 REFERENCES DEPT);
Transformation
By Ram Reddy
Types of Transformation
1. Passive transformation
1. Operates on one row of data at a time AND 2. Cannot change the number of rows on the data flow 3. Example: Expression,Lookup,Stored Procedure transformation
2. Active transformation
1. Can operate on groups of data rows AND/OR 2. Can change the number of rows on the data flow 3. Examples: Aggregator, Filter, Source Qualifier
#209 Nilagiri Block Adithya Enclave Ameerpet,HYD @8801408841, 8790998182 , , ,
Example
Passive
Tax=Sal *0.1
14 Rows
14 Rows
Active
Deptno=10
14 Rows
5 Rows
Types of Transformation
1. Connected transformation
1. 2. 3. Is connected in mapping All active and passive can use as connected Connected transformation receives multiple input ports and can provides multiple outputs Is unconnected in mapping Stored and lookup transformations only can use as unconnected unconnected transformation receives multiple input ports and provides only one output
2. Unconnected transformation
1. 2. 3.
Example
Connected
Unconnected
Expression Transformation
By Ram Reddy
Expression Transformation
Perform calculations using non-aggregate functions (row level)
Ports Mixed Variables allowed Create expression in an output or variable port Usage Perform majority of data manipulation
An expression formula is a calculation or conditional statement for a specific port in a transformation Performs calculation based on ports, functions, operators, variables, constants and return values from other transformations
Expression Editor
Expression Validation
The Validate or OK button in the Expression Editor will:
Parse the current expression
Example Expression
Example Expression
Tasks
Are two types 1. Reusable Tasks 2. Non - Reusable Tasks
1. Reusable Tasks
Three types of reusable Tasks
Session Set of instructions to execute a specific Mapping Command Specific shell commands to run during any Workflow Email Sends email during the Workflow
Reusable Tasks
Use the Task Developer to create reusable tasks These tasks will then appear in the Navigator and can be dragged and dropped into any workflow
Reusable
Non-reusable
1. Session Tasks
Session Task
Server instructions to run the logic of ONE specific mapping e.g. source and target data location specifications, memory allocation, optional Mapping overrides, scheduling, processing and load instructions Becomes a component of a Workflow (or Work let)
Or Select menu Tasks | Create and select Session from the drop -down menu
Or Select menu Tasks | Create and select Session from the drop -down menu
Set properties
Session Configuration
Define properties to be reusable across different sessions Defined at folder level Must have one of these tools open in order to access
2. Command Task
Specify one or more Unix shell or DOS commands to run during the Workflow Runs in the Informatica Server (UNIX or Windows) environment Command task status (successful completion or failure) is held in the pre-defined task variable $command_task_name.STATUS Each Command Task shell command can execute before the Session begins or after the Informatica Server executes a Session
Command Task
Specify one (or more) Unix shell or DOS (NT, Win2000) commands to run at a specific point in the workflow Becomes a component of a workflow (or worklet) If created in the Task Developer, the Command task is reusable If created in the Workflow Designer, the Command task is not reusable Commands can also be invoked under the Components tab of a Session task to run pre- or post session -
Command Task(Example)
Command Task(Example)
3. Email Task
Configure to have the Informatica Server to send email at any point in the Workflow Becomes a component in a Workflow (or Worklet) If configured in the Task Developer, the Email Task is reusable (optional) Emails can also be invoked under the Components tab of a Session task to run pre- or post-session
Non-Reusable Tasks
Non-Reusable Tasks
Six additional Tasks are available in the Workflow Designer
Decision Assignment Timer
1. Decision Task
Specifies a condition to be evaluated in the Workflow Use the Decision Task in branches of a Workflow Use decision task instead of multiple link conditions .
Decision Task
The Decision task has a predefined variable called $Decision_task_name.condition that represents the result of the decision condition. The Integration Service evaluates the condition in the Decision task and sets the predefined condition variable to True (1) or False (0).
Decision Task(Example)
2. Assignment Task
Assigns a value to a Workflow Variable Variables are defined in the Workflow object
Assignment Task
3. Timer Task
Waits for a specified period of time to execute the next Task
General Tab Timer Tab
Relative time
v We instruct the Power Center Server to wait for a specified period of time after the Timer task, the parent workflow, or the top-level workflow starts
Timer Task(Example)
4. Control Task
Stop or ABORT the Workflow
Properties Tab General Tab
Control Task(Example)
General Tab
Properties Tab
Filter Transformation
By Ram Reddy
Filter Transformation
1. Is a condition created with following elements
1. Port or column 2. Operator 3. Operand Example Deptno
1
=
2
10
3
Filter Transformation
1. Is Active 2. Is connected 3. Use the Filter transformation to filter out rows in a mapping 4. Use to cleanse data (filter unwanted data) 5. The naming convention for a Filter transformation is FIL_TransformationName 6. Condition to reject rows of records which have FIRST_NAME null IIF(ISNULL(FIRST_NAME),FALSE,TRUE)
#209 Nilagiri Block Adithya Enclave Ameerpet,HYD @8801408841, 8790998182 , , ,
Filter Transformation
Drops rows conditionally
Ports All input / output Specify a Filter condition Usage Filter rows from input flow
Example Filter
Worklets
An object representing a set or grouping of Tasks Can contain any Task available in the Workflow Manager Worklets expand and execute inside a Workflow A Workflow which contains a Worklet is called the parent Workflow Worklets CAN be nested Reusable Worklets create in the Worklet Designer Non-reusable Worklets create in the Workflow Designer
Re-usable Worklet
In the Worklet Designer, select Worklets | Create
Non-Reusable Worklet
1. Create worklet task in Workflow Designer Right-click on new worklet and select Open Worklet Workspace switches to Worklet Designer 2.
3.
Debugger
By the end of this section you will be familiar with: Creating a Debug Session Debugger windows and indicators Debugger functionality and options Viewing data with the Debugger Setting and using Breakpoints Tips for using the Debugger
Debugger Features
Wizard driven tool that runs a test session View source / target data View transformation data Set breakpoints and evaluate expressions Initialize variables Manually change variable values Data can be loaded or discarded Debug environment can be saved for later use
Debugger Interface
Edit Breakpoints Debugger Mode indicator Solid yellow arrow is current transformation indicator
Set Breakpoints
1. Edit breakpoint 2. Choose global or specific transformation 3. Choose to break on data condition or error. Optionally skip rows. 4. Add breakpoint(s) 5. Add data conditions
Debugger Tips
Server must be running before starting a Debug Session When the Debugger is started, a spinning icon displays. Spinning stops when the Debugger Server is ready The flashing yellow/green arrow points to the current active Source Qualifier. The solid yellow arrow points to the current Transformation instance
Next Instance proceeds a single step at a time; one row moves from transformation to transformation Step to Instance examines one transformation at a time, following successive rows through the same transformation
#209 Nilagiri Block Adithya Enclave Ameerpet,HYD @8801408841, 8790998182 , , ,
Sorter Transformation
By Ram Reddy
Sorter Transformation
Can sort data from relational tables or flat files Sort takes place on the Informatica Server machine Multiple sort keys are supported The Sorter transformation is often more efficient than a sort performed on a database with an ORDER BY clause Is useful to eliminate duplicate records so it is called as active transformation
Sorter Transformation
Sorts data from any source, at any point in a data flow
Sort Keys
Ports Input/Output Define one or more sort keys Define sort order for each key Example of Usage Sort data before Aggregator to improve performance
Sort Order
Sorter Properties
Ensure sufficient memory is available on the Informatica Server (else Session Task will fail)
Aggregator Transformation
By Ram Reddy
Aggregator Transformation
By the end of this section you will be familiar with: Basic Aggregator functionality Creating subtotals with the Aggregator Aggregator expressions Aggregator properties Using sorted data
Aggregator Transformation
Performs aggregate calculations
Ports Mixed I/O ports allowed Variable ports allowed Group By allowed Create expressions in variable and output ports Usage Standard aggregations
Aggregate Expressions
Aggregate functions are supported only in the Aggregator Transformation
Aggregator Functions
AVG COUNT FIRST LAST MAX MEDIAN MIN PERCENTILE STDDEV SUM VARIANCE
Return summary values for non-null data in selected ports Use only in Aggregator transformations Use in output ports only Calculate a single value (and row) for all records in a group Only one aggregate function can be nested within an aggregate function Conditional statements can be used with these functions
Aggregator Properties
Sorted Input Property
Instructs the Aggregator to expect the data to be sorted Set Aggregator cache sizes for Informatica Server machine
No rows are released from Aggregator until all rows are aggregated
Each separate group (one row) is released as soon as the last row in the group is aggregated
Aggregate cache. The Integration Service stores data in the aggregate cache until it completes aggregate calculations. It stores group values in an index cache and row data in the data cache. Aggregate expression. Enter an expression in an output port. The expression can include non-aggregate expressions and conditional clauses Group by port. Indicate how to create groups. The port can be any input, input/output, output, or variable port. When grouping data, the Aggregator transformation outputs the last row of each group unless otherwise specified. Sorted input. Select this option to improve session performance. To use sorted input, you must pass data to the Aggregator transformation sorted by group by port, in ascending or descending order.
Monitoring Workflows
Monitoring Workflows
By the end of this section you will be familiar with: The Workflow Monitor GUI interface Monitoring views Server monitoring modes Filtering displayed items Actions initiated from the Workflow Monitor Truncating Monitor Logs
Workflow Monitor
The Workflow Monitor is the tool for monitoring Workflows and Tasks Choose between two views:
Task view
Monitoring Operations
Perform operations in the Workflow Monitor
Stop, Abort, or Restart a Task, Workflow or Worklet Resume a suspended Workflow after a failed Task is corrected Reschedule or Unschedule a Workflow
Status Bar
Right-click on Session to retrieve the Session Log (from the Server to the local PC Client)
Filter Toolbar
Select type of tasks to filter Select servers to filter Filter tasks by specified criteria Display recent runs
Repository Manager Repository Manager s Truncate Log option clears the Workflow Monitor logs
Joiner Transformation
By Ram Reddy
Joiner Transformation
By the end of this section you will be familiar with: When to join in Source Qualifier and when in Joiner transformation Homogeneous joins Heterogeneous joins Joiner properties Joiner conditions Nested joins
Joiner Transformation
Performs heterogeneous joins on different data flows
Active Transformation Ports All input or input / output M denotes port comes from master source Examples Join two flat files Join two tables from different databases Join a flat file with a relational table
Joiner Conditions
Joiner Properties
Join types:
Normal (inner) Master outer Detail outer Full outer Set Joiner Caches
Joiner can accept sorted data (configure the join condition to use the sort origin ports)
#209 Nilagiri Block Adithya Enclave Ameerpet,HYD @8801408841, 8790998182 , , ,
Nested Joins
Used to join three or more heterogeneous sources
Performance Tuning
By Ram Reddy
Performance Tuning
First step in performance tuning is to identify the performance bottleneck in the following order: Target Source Mapping Session System The most common performance bottleneck occurs when the Informatica Server writes to a target database.
Target Bottlenecks
Identifying
A target bottleneck can be identified by configuring the session to write to a flat file target.
Optimizing
Dropping Indexes and Key Constraints before loading. Increasing commit intervals. Use of Bulk Loading / External Loading.
Source Bottlenecks
Identifying Add a filter condition after Source qualifier to false so that no data is processed past the filter transformation. If the time it takes to run the new session remains about the same, then there is a source bottleneck. In a test mapping remove all the transformations and if the performance is similar, then there is a source bottleneck. Optimizing Optimizing the Query by using hints. Use Informatica Conditional Filters if the source system lacks indexes.
# 209 Nilagiri Block Adithya Enclave Ameerpet,HYD @8801408841,8790998182 , , ,
Identifying If there is no source bottleneck, add a Filter transformation in the mapping before each target definition. Set the filter condition to false so that no data is loaded into the target tables. If the time it takes to run the new session is the same as the original session, there is a mapping bottleneck. Optimizing Configure for Single-Pass reading Avoid unnecessary data type conversions. Avoid database reject errors. Use Shared Cache / Persistent Cache
# 209 Nilagiri Block Adithya Enclave Ameerpet,HYD @8801408841,8790998182 , , ,
Mapping Bottlenecks
Session Bottlenecks
Identifying
If there is no source, Target or Mapping bottleneck, then there may be a session bottleneck. Use Collect Performance Details. Any value other than zero in the readfromdisk and writetodisk counters for Aggregator, Joiner, or Rank transformations indicate a session bottleneck. Low (0-20%) BufferInput_efficiency and BufferOutput_efficiency counter values also indicate a session bottleneck.
Optimizing
Increase the number of partitions. Tune session parameters. DTM Buffer Size (6M 128M) Buffer Block Size (4K 128K) Data (2M 24 M )/ Index (1M -12M) Cache Size Use incremental Aggregation if possible.
# 209 Nilagiri Block Adithya Enclave Ameerpet,HYD @8801408841,8790998182 , , ,
System Bottlenecks
Identifying If there is no source, Target, Mapping or Session bottleneck, then there may be a system bottleneck. Use system tools to monitor CPU usage, memory usage, and paging. On Windows :- Task Manager On Unix Systems tools like sar, iostat. For E.g. sar u (%usage on user, idle time, i/o waiting time) Optimizing Improve network speed. Improve CPU performance Check hard disks on related machines Reduce Paging
Incremental Aggregation
First Run creates idx and dat files. Second Run performs the following actions: For each i/p record, the Server checks historical information in the index file for a corresponding group, then: If it finds a corresponding group, it performs the aggregate operation incrementally, using the aggregate data for that group, and saves the incremental change If it does not find a corresponding group, it creates a new group and saves the record data When writing to the target Informatica Server Updates modified aggregate groups in the target Inserts new aggregate data Deletes removed aggregate data Ignores unchanged aggregate data Saves209 Nilagiri Block Adithyadata in the index and @8801408841,8790998182 modified aggregate Enclave Ameerpet,HYD data files # , , ,
Incremental Aggregation
You can find options for incremental aggregation on the Transformations tab in the session properties The Server Manager displays a warning indicating the Informatica Server overwrites the existing cache and a reminder to clear this option after running the session
# 209 Nilagiri Block Adithya Enclave Ameerpet,HYD @8801408841,8790998182 , , ,
Commit Points
A commit interval is the interval at which the Informatica Server commits data to relational targets during a session The commit point can be a factor of the commit interval, the commit interval type, and the size of the buffer blocks The commit interval is the number of rows you want to use as a basis for the commit point The commit interval type is the type of rows that you want to use as a basis for the commit point Can choose between the following types of commit interval Target-based commit Source-based commit During a source-based commit session, the Informatica Server commits data to the target based on the number of rows from an active source in a single pipeline
Commit Points
During a target-based commit session, the Informatica Server continues to fill the writer buffer after it reaches the commit interval When the buffer block is filled, the Informatica Server issues a commit command As a result, the amount of data committed at the commit point generally exceeds the commit interval
Commit Points
During a source-based commit session, the Informatica Server commits data to the target based on the number of rows from an active source in a single pipeline These rows are referred to as source rows A pipeline consists of a source qualifier and all the transformations and targets that receive data from the source qualifier An active source can be any of the following active transformations: Advanced External Procedure Source Qualifier Normalizer Aggregator Joiner Rank Mapplet, if it contains one of the above transformations
# 209 Nilagiri Block Adithya Enclave Ameerpet,HYD @8801408841,8790998182 , , ,
Commit Points
When the Informatica Server runs a source-based commit session, it identifies the active source for each pipeline in the mapping The Informatica Server generates a commit row from the active source at every commit interval When each target in the pipeline receives the commit row, the Informatica Server performs the commit
# 209 Nilagiri Block Adithya Enclave Ameerpet,HYD @8801408841,8790998182 , , ,
Commit Points
Constraint-based Loading
PK1
FK1 PK2
FK2
To maintain referential integrity, primary keys must be loaded before their corresponding foreign keys here in the order Target1, Target2, Target 3
# 209 Nilagiri Block Adithya Enclave Ameerpet,HYD @8801408841,8790998182 , , ,
Active source
Active transformation that generates rows Cannot match an output row with a distinct input row Examples: Source Qualifier, Aggregator, Joiner, Sorter (The Filter is NOT an active source)
Active group
Group of targets in a mapping being fed by the same active source
Example 1
With only one Active source, rows for Targets1, 2, and 3 will be loaded properly and maintain referential integrity
PK1
Example 2
With two Active sources, it is not possible to control whether rows for Target3 will be loaded before or after those for Target2
Ports
All input/output Convert datatypes For relational sources: Modify SQL statement User Defined Join Source Filter Sorted ports Select DISTINCT Pre/Post SQL
Usage
Lookup Transformation
By Ram Reddy
Lookup Transformation
Can use in two ways
1. Connected 2. Un connected
Lookup Transformation
By the end of this section you will be familiar with: Lookup principles Lookup properties Lookup conditions Lookup techniques Caching considerations Persistent caches
Return value(s)
Lookup Transformation
Looks up values in a database table or flat file and provides data to other components in a mapping
Ports Mixed L denotes Lookup port R denotes port used as a return value (unconnected Lookup only see later) Specify the Lookup Condition Usage Get related values Verify if records exists or if data has changed
Lookup Conditions
Lookup Properties
Lookup table name Lookup condition Native database connection object name Source type: Database or Flat File
Policy on multiple match: Use first value Use last value Report error
Lookup Caching
Caching can significantly impact performance Cached
Lookup table data is cached locally on the Server Mapping rows are looked up against the cache Only one SQL SELECT is needed
Uncached
Each Mapping row needs one SQL SELECT Rule Of Thumb: Cache if the number (and size) of records in the Lookup table is small relative to the number of mapping rows requiring the lookup
Persistent Caches
By default, Lookup caches are not persistent; when the session completes, the cache is erased Cache can be made persistent with the Lookup properties When Session completes, the persistent cache is stored on the server hard disk The next time Session runs, cached data is loaded fully or partially into RAM and reused A named persistent cache may be shared by different sessions Can improve performance, but stale data may pose a problem
Set prefix for persistent cache file name Reload persistent cache
#209 Nilagiri Block Adithya Enclave Ameerpet,HYD @8801408841, 8790998182 , , ,
2. Unconnected Lookups
Unconnected Lookups
By the end of this section you will know: Unconnected Lookup technique Unconnected Lookup functionality Difference from Connected Lookup
Unconnected Lookup
Physically unconnected from other transformations NO data flow arrows leading to or from an unconnected Lookup Lookup data is called from the point in the Mapping that needs it Lookup function can be set within any transformation that supports expressions
Function in the Aggregator calls the unconnected Lookup
IIF ( ISNULL(customer_id),:lkp.MYLOOKUP(order_no))
Lookup function
Condition is evaluated for each row but Lookup function is called only if condition satisfied
Must check a Return port in the Ports tab, else fails at runtime
Part of the mapping data flow Returns multiple values (by linking output ports to another transformation) Executed for every record passing through the transformation More visible, shows where the lookup values are used Default values are used
Separate from the mapping data flow Returns one value- by checking the Return (R) port option for the output port that provides the return value Only executed when the lookup function is called Less visible, as the lookup is called from an expression within another transformation Default values are ignored
Business Challenge
When you design a data warehouse, you need to decide what type of information to store in targets. As part of the target table design, you need to determine whether to maintain all the historic data or just the most recent changes.
Update Levels
Within a session: When you configure a session, you can instruct the Integration Service to either treat all rows in the same way (for example, treat all rows as inserts), or use instructions coded into the session mapping to flag rows for different database operations. Within a mapping: Within a mapping, you use the Update Strategy transformation to flag rows for insert, delete, update, or reject.
0 1 2 3
Ports All input / output Specify the Update Strategy Expression IIF or DECODE logic determines how to handle the record Example Updating Slowly Changing Dimensions
3. Update else Insert. Update the row if it exists. Otherwise, insert it. 4. Truncate table. Select this option to truncate the target table before loading data.
# 209 Nilagiri Block Adithya Enclave Ameerpet,HYD @8801408841,8790998182 , , ,
Normalizer Transformation
By Ram Reddy
Normalizer Transformation
Is Active and connected transformation Is useful to read data from Cobol files Is associated with Cobol file Is useful to convert a single input record into multiple output records Is useful to generate GCID (Generated Column id )
Example
Year 2012 Year Account Q1 1000 Q2 Q3 Q4 2012 2012 2012 2012 Saving 2000 3000 4000 Account Month Saving Saving Saving Saving 1 2 3 4 Amount 1000 2000 3000 4000
Normalizer
Creating Keys
1. You can create primary or foreign key values with the Sequence Generator transformation by connecting the NEXTVAL port to a target or downstream transformation. 2. You can use a range of values from 1 to 9,223,372,036,854,775,807 with the smallest interval of 1. 3. Called as surrogate keys
# 209 Nilagiri Block Adithya Enclave Ameerpet,HYD @8801408841,8790998182 , , ,
Ports Two predefined output ports, NEXTVAL and CURRVAL No input ports allowed Usage Generate sequence numbers Shareable across mappings
Union Transformation
By Ram Reddy
Union Transformation
Useful to combine two similar structures (Number of columns and corresponding data types must be same in two source pipelines Types of Groups
1. 2. Input (two or more) Output
Union Transformation
Is active transformation Is useful to combine multiple similar input pipelines into one output pipe line Supports heterogeneous data sources It works similar to UNION ALL function in data base
MFD
Set datatype User-defined names Set aggregation type Set optional initial value
Parameter Files
You can specify a parameter file for a session in the session editor File extension must be .prm or .txt Parameter file contains folder.session or folder.WF:workflow name.ST:session name and initializes each parameter and variable for that session. For example:
[Production.s_m_MonthlyCalculations] $$State=MA $$Time=10/1/2000 00:00:00 $InputFile1=sales.txt $DBConnection_target=sales $PMSessionLogFile=D:/session logs/firstrun.txt
#
By Ram Reddy
Mapplets
Mapplets
By the end of this section you will be familiar with: Mapplet Designer Mapplet advantages Mapplet types Mapplet rules Active and Passive Mapplets Mapplet Parameters and Variables
Mapplet Designer
Mapplet Advantages
A mapplet is a reusable object that you create in the Mapplet Designer Useful for repetitive tasks / logic Represents a set of transformations Mapplets are reusable Use an instance of a Mapplet in a Mapping Changes to a Mapplet are inherited by all instances Server expands the Mapplet at runtime
Mapplet Types
Mapplets are two types 1. Active 2. Passive A mapplet contains at least one active transformation then it is called as Active Mapplet A mapplet contains all passive transformation then it is called as Passive Mapplet
Unsupported Transformations
Do not use the following in a mapplet: XML source definitions Normalizer transformation Pre or post stored procedure transformation COBOL Sources Target definitions Other mapplets(nested mapplets)
Passive Transformation Connected Ports Output ports only Usage Only those ports connected from an Input transformation to another transformation will display in the resulting Mapplet
#
Transformation
Connecting the same port to more than one transformation is disallowed Pass to an Expression transformation first
Transformation
Resulting Mapplet HAS input ports When used in a Mapping, the Mapplet may occur at any point in mid-flow
Mapplet
Source Qualifier
Resulting Mapplet has no input ports When used in a Mapping, the Mapplet is the first object in the data flow
#
Mapplet
Notes
You can expand the mapplet in the Mapping Designer by selecting it and clicking Mappings > Expand If you use a Sequence Generator transformation, you must use a reusable Sequence Generator transformation. If you use a Stored Procedure transformation, you must configure the Stored Procedure Type to be Normal.
Reusable Transformations
By Ram Reddy
Reusable Transformations
By the end of this section you will be familiar with: Reusable transformation rules Different ways to develop reusable transformations Transformation Developer Promoting transformations to reusable
Reusable Transformations
Define once, reuse many times Reusable Transformations
Can be a copy or a shortcut Edit Ports only in Transformation Developer Can edit Properties in the mapping Instances dynamically inherit changes Caution: changing reusable transformationscan invalidate mappings Note: Source Qualifier transformations cannot be made reusable
Transformation Developer
Make a transformation reusable from the outset, or test it in a mapping first
Reusable transformations