GoldenGate An Overview

Agenda
1 GoldenGate - Introduction 2 Architecture 3 Implementation 4 Conclusion

Introduction

GoldenGate - Introduction
GoldenGate Software Inc. is a leading provider of high availability and real-time data integration solutions. GoldenGate helps mission-critical systems maintain 24/7 operations to meet end-user demand for continuous access. GoldenGate also helps organizations to immediately distribute critical data across the enterprise, to enable timely and accurate decision making. In short, GoldenGate solutions enable real-time access to real-time information. GoldenGate solutions are powered by GoldenGates Transactional Data Management (TDM) platform, which provides continuous capture, routing, transformation, and delivery of transactional data between heterogeneous systems in real time and with minimal overhead.

GoldenGate Introduction (Contd)

SOLUTIONS FOR HIGH AVAILABILITY AND DISASTER TOLERANCE

GoldenGate High Availability and Disaster Tolerance solutions eliminate downtime caused by unplanned and planned outages and boost the performance and scalability of systems burdened by the tremendous growth of data or usage demands.
GoldenGate Live Standby significantly improves recovery time for business-critical systems. GoldenGate Zero-Downtime Operations enables uninterrupted business operations during necessary system upgrade, migration, and maintenance activities. GoldenGate Active-Active improves performance and reliability for two or more databases by enabling workload balancing. Database Tiering allows companies to ensure the highest performance of their production systems while still easily supporting necessary read-only activities.

GoldenGate Introduction (Contd)

SOLUTIONS FOR REAL-TIME DATA INTEGRATION

GoldenGate Real-Time Data Integration solutions provide real-time change data to data warehouses, operational data stores, reporting systems, or other OLTP-supporting databases without batch windows and with minimal performance impact.
GoldenGate Real-Time Data Warehousing enables continuous, real-time capture, transformation, and delivery of the most recent change data between OLTP systems and the data warehouse. It can also integrate with existing ETL solutions. GoldenGate Live Reporting off-loads reporting activity from the production database to a lower-cost secondary system, where the most current data is always available to support real-time reporting needs. GoldenGate Transactional Data Integration offers point-to-point data integration between OLTP systems in real time and with minimal impact.

GoldenGate Introduction (Contd)

Goldengate TRANSACTIONAL DATA MANAGEMENT (TDM)
GoldenGate TDM software moves transactional data between heterogeneous systems in real time, with transactional integrity.
Key Features: Real-Time Data As new transactions come in, the data is immediately captured, transformed (as needed), and delivered to other systems with only sub-second latency. Continuous Availability GoldenGate works without requiring system interruption or outage windows. Heterogeneous Support The underlying databases and platforms for critical applications can be heterogeneous, allowing significant flexibility for the IT department. High Performance with Low Impact GoldenGate can support the movement of thousands of transactions per second yet imposes negligible impact on the source and target systems.

Architecture

Architecture

Sources Supported
Databases: Oracle DB2 Microsoft Sql Server Sybase ASE Ingres Teradata Enscribe SQL/MP SQL/MX Platforms: HP NonStop HP-UX HP TRU64 HP OpenVMS Windows 2000, 2003, XP Linux Sun Solaris IBM AIX IBM z/OS

Architecture (Contd)
Targets Supported
Databases: Oracle DB2 Microsoft Sql Server Sybase ASE Ingres Teradata Enscribe SQL/MP SQL/MX MySQL HP Neoview Netezza Greenplum JMS Message Bus ETL Servers Any ODBC compatible database Platforms: HP NonStop HP-UX HP TRU64 HP OpenVMS Windows 2000, 2003, XP Linux Sun Solaris IBM AIX IBM z/OS

Architecture (Contd)
The following are the components & processes involved in the GoldenGate Capture (Extract)
Capture module reads the transaction logs (redo or archive logs) of the desired source database. Captures committed transactions - insert, update & delete operations - and maintains transaction integrity. Immediately writes the transactions to GoldenGate external trail file for distribution to targets Process efficiently and with low impact - does not create additional work in the database or create triggers Offers selective capture and delivery capability: Advanced filtering at the database/schema, table, row or column level or based on operation type, matched patterns and value thresholds

Source/Target Trail Files

Trail Files contain the changed data operations in a transportable, platformindependent, universal data format. Exist outside of the databases to minimize overhead and enable improved reliability and heterogeneity Help to checkpoint data, to ensure data accuracy and completeness between source and target systems

Architecture (Contd)
Data Pump (Capture Once, Apply Many)
The GoldenGate Pump reads the trail file and distributes the data to the designated target database Multiple pumps may be configured to send the data to multiple targets, even different databases (ie. capture from Oracle, apply to Teradata & Sql Server)

Routing
GoldenGate send data over TCP/IP - this means no distance constraints between source and target systems! Encryption may be applied Additional compression may be applied as desired Can send thousands of transaction operations per second (TPS) Extremely efficient, low-bandwidth implementation

Delivery (Replicat)
GoldenGate Delivery reads changed data transactions from the Trail File Continuously applies changed data to the target database using the native Sql for that RDBMS Ensures transaction integrity by applying transactions in the same order that they were committed on the source system

Architecture (Contd)
Transaction commit boundaries are preserved throughout each step of the data movement Delivery also supported to non-RDBMS targets via flat file integration Supports high-volume environments while keeping source and target "in sync" All of this happens with only sub-second latency!!

Manager
Manager is GoldenGates control process. Manager must be running on each system involved in GoldenGate synchronization. Manager performs the following functions: Monitor and restart GoldenGate processes. Issue threshold reports, for example when Extract processing slows down or when synchronization latency increases. Maintain trail files and logs. Allocate data storage space. Report errors and events. Receive and route user requests from the user interface.

Architecture (Contd)
Collector
Collector is a process that runs in the background on the target system to receive data sent across the TCP/IP network and write it to the trail or extract file. Typically, Manager starts Collector automatically

Implementation

Implementation
GoldenGate can be installed and configured quickly and easily in just four easy Steps: 1. Prepare the environment
2. Configure Change Capture
3. Initial Data Load 4. Configure Change Delivery

Implementation - Prepare the environment

Step 1: Set up each system

Install GoldenGate
Start GoldenGate Manager Generate source definitions for heterogeneous configurations Add supplemental log data from Oracle or MS SQL log-based solution Add loop-detection for bi-directional configuration

Implementation Configure Change

Capture
Step 2: Capture changes to the source

Add Extract checkpoint into the transaction log

Configure Extract parameter file Define GoldenGate trails Start Extract

Implementation Initial Data Load

Step 3: Load in the starting data

Configure Extract Parameter File

Configure Replicat Parameter File Start Extract Start Replicat

Implementation Configure Change

Delivery
Step 4: Apply changes to the target database

Add Replicat checkpoint

Configure Replicat parameter file Start Replicat

Installation Easy Steps

Unix/Linux
1. 2. FTP the file to the UNIX system in binary mode, and place it in the directory where you want GoldenGate to be installed. Extract the tar file (use the gzip or tar options appropriate for your system). Syntax: /ggs> gzip -dc filename.tar.gz | tar -xvf All GoldenGate files are placed in the current directory. Download the GoldenGate password file in ASCII mode and save it with a file name of pw (lowercase, no file extension) in the same directory where you installed the GoldenGate software. From the GoldenGate directory, run the GGSCI program. GGSCI In GGSCI, issue the following command. CREATE SUBDIRS Issue the following command to exit GGSCI. EXIT Configure the Manager process

3.

4. 5. 6. 7.

Installation Easy Steps (Cond)

Windows
1. 2. Unzip the downloaded file(s) using PKUNZIP or WinZip Move the files in binary mode to a folder (for eg: C:\GGS) on the drive where you want to install GoldenGate. Do not install GoldenGate into a folder that contains spaces in its name, for example GoldenGate Software. The application references path names, and the operating system does not support path names that contain spaces, whether or not they are within quotes. Download the GoldenGate password file in ASCII mode and save it with a file name of pw (lowercase, no file extension) in the GoldenGate folder (for eg: C:\GGS). From the GoldenGate folder, run the GGSCI program GGSCI In GGSCI, issue the following command. CREATE SUBDIRS Issue the following command to add service & event messages to the system registry C:\GGS> INSTALL ADDSERVICE ADDEVENTS USER <specification> PASSWORD <password> Issue the following command to exit GGSCI EXIT

3. 4. 5. 6.

7.

By default, the errors logged are generic. To enable the display of more specific error content, copy the category.dll & ggsmsg.dll files from the GoldenGate installation directory to the SYSTEM32 directory. The enhanced output

Installation Easy Steps (Cond)

Configuring Manager
To configure Manager, create a parameter file. From the GoldenGate directory, run the ggsci program. This is the GoldenGate Software Command Interface, commonly known as GGSCI. In GGSCI, issue the following command. EDIT PARAMS MGR The only required parameter for Manager is PORT, which defines the port number on which Manager runs on the local system. The default port is 7809. You must specify either the default port or another port. This must bean unreserved, unrestricted port. GGSCI uses the port to request Manager to start processes. The Extract process uses the port to request Manager to start a remote Collector process or an initial-load Replicat process. PORT <port_number> Note: The port number also must be specified with the MGRPORT argument of the Extract parameter RMTHOST.

To run Manager from GGSCI

From the GoldenGate directory, run GGSCI. In GGSCI, issue the following command. START MANAGER

GoldenGate Parameter Files

Using GoldenGate parameter files
Most GoldenGate functionality is controlled by means of parameters specified in parameter files. A parameter file is an ASCII file that is read by an associated process. GoldenGate uses two types of parameter files: a GLOBALS file and runtime parameter files.

Overview of the GLOBALS file

The GLOBALS file stores parameters that relate to the GoldenGate instance as a whole. This is in contrast to runtime parameters, which are coupled with a specific process such as Extract. Parameters in the GLOBALS file apply to all processes in the GoldenGate instance, but can be overridden by specific process parameters. Once set, GLOBALS parameters are rarely changed, and there are far fewer of them than runtime parameters. A GLOBALS parameter file is required only in certain circumstances and, when used, must be created from the command shell before starting any GoldenGate processes, including GGSCI. The GGSCI program reads the GLOBALS file and passes the parameters to processes that need them. To create a GLOBALS file 1. From the GoldenGate home location, run GGSCI and enter the following command (or you can simply open a file in an ASCII text editor).

GoldenGate Parameter Files

3. Save the file. If you used a text editor, save the file with the name GLOBALS (all capital letters on UNIX-based systems) without a file extension in the root GoldenGate directory (where the binaries are stored). If you created the file in GGSCI, the file is saved that way automatically. Do not move this file. Exit GGSCI and start a new session before issuing commands or starting processes that reference the GLOBALS file.

Overview of runtime parameters Runtime parameters give you control over the various aspects of GoldenGate synchronization, such as: Data selection, mapping, transformation, and replication DDL selection, mapping, and replication (where supported) Error resolution Logging Status and error reporting System resource usage Startup and runtime behavior Note: There can be only one active parameter file for the Manager process or an Extract or Replicat group. There are two types of parameters: global (not to be confused with GLOBALS parameters) and tablespecific.

GoldenGate Parameter Files

MAP parameter statements that specify the database objects whose data or DDL is to be processed, and it should be listed only once in the file. When listed more than once, only the last instance of the parameter is active. All other instances are ignored. Table-specific parameters enable you to create different processing rules for different sets of tables that are specified with TABLE or MAP parameter statements. GETINSERTS and IGNOREINSERTS in Figure 5 are examples of table-specific parameters. Table-specific parameters take effect in the order that each parameter is listed in the file.

The following are examples of basic parameter files for Extract and Replicat. Sample Extract parameter file EXTRACT capt USERID etluser, PASSWORD ********* DISCARDFILE /ggs/capt.dsc, PURGE RMTHOST sysb, MGRPORT 7809 RMTTRAIL /ggs/dirdat/aa TABLE fin.*; TABLE sales.*;

GoldenGate Parameter Files

Sample Replicat parameter file REPLICAT deliv USERID etluser, PASSWORD **** SOURCEDEFS /ggs/dirdef/defs DISCARDFILE /ggs/deliv.dsc, PURGE GETINSERTS MAP fin.account, TARGET fin.acctab, COLMAP (account = acct, balance = bal, branch = branch); MAP fin.teller, TARGET fin.telltab, WHERE (branch = NY); IGNOREINSERTS MAP fin.teller, TARGET fin.telltab, WHERE (branch = LA); Changing a parameter file You can change parameter settings after processing has started. You can change any Manager parameter except the port number while the Manager process is running; to change the port number, you must stop

GoldenGate Parameter Files

running, changes to parameter settings can have unpredictable and adverse consequences, especially if you are adding tables or changing mapping or filtering rules. To change Manager parameters 1. If changing the port number, stop the Manager process by using the STOP MANAGER command in GGSCI. If you are not changing the port, do not stop Manager. STOP MANAGER 2. Open the parameter file by using a text editor or the EDIT PARAMS command in GGSCI. EDIT PARAMS mgr 3. Make the edits, and then save the file. 4. Do one of the following: Refresh the Manager process if it is still running, REFRESH MANAGER Start Manager if it is not running. START MANAGER To change Extract or Replicat parameters 1. Stop the process. STOP {EXTRACT | REPLICAT} <group name> 2. Open the parameter file by using a text editor or the EDIT PARAMS commanding GGSCI. EDIT PARAMS mgr 3. Make the edits, and then save the file. 4. Start the process. START {EXTRACT | REPLICAT} <group name>

Initial Data Load

Initial Data Load Three Methods GoldenGate provides three methods known as Queue Data, Direct Load and Direct Bulk Load Queue Data: Queue Data approach provides the ability to capture data directly from the source tables, and output the data to GoldenGate Trails. The trails can be formatted to ASCII so that it can be loaded by native bulk utilities such as DTS, BCP or SQL*Loader Direct Load: Direct Load approach provides the ability to capture data directly from the source tables and send the data in a large block to the Replicat process. The Manager process will dynamically start the Replicat process. Direct Bulk Load: Direct Load approach provides the ability to capture data directly from the source tables and send the data in a large block to the Replicat process which then communicates directly to Sql*Loaders using an API. The Manager process will dynamically start the Replicat process. This is the fastest approach. Initial Data Load Three Step Process Step 1: Start change capture

Initial Data Load

Define GoldenGate queues 2. Start Extract and verify its running Step 2: Perform initial data load using any of the 3 methods discussed above and follow the below steps 1. Configure Extract to read directly from the source database via the SOURCEISTABLE parameter 2. Configure Replicat with the parameters SPECIALRUN 3. Start the initial data Extract Step 3: Start the delivery of changes When the initial data load completes 1. Configure Replicat on the target system with the HANDLECOLLISIONS and END parameters 2. The END parameter is the time recorded for the completion of the source file Extract process 3. Start the Replicat delivery process 4. When the Replicat process stops, remove the END and HANDLECOLLISIONS parameters 5. Start the Replicat process for incremental data synchronization Queue Data Example parameter file Example Extract parameter file SOURCEISTABLE USERID etluser, PASSWORD ********* RMTHOST targethost, MGRPORT 7809 RMTFILE /ggs/account.dat, purge TABLE SALES.ACCOUNT;

Initial Data Load

Example Replicat parameter file SPECIALRUN END RUNTIME USERID tgtuser, PASSWORD ********* ASSUMETARGETDEFS EXTFILE /ggs/account.dat MAP SALES.ACCOUNT, TABLE SALES.ACCOUNT; Queue Data Execution To execute the Extract and Replicat process, execute the EXTRACT and REPLICAT programs from the command shell. Windows C:\GGS> extract paramfile dirprm\initext.prm C:\GGS> replicat paramfile dirprm\initrep.prm Unix $ /ggs> extract paramfile dirprm/initext.prm $ /ggs> replicat paramfile dirprm/initrep.prm

Initial Data Load

Direct Load Example parameter file Example Extract parameter file EXTRACT INITEXT USERID etluser, PASSWORD ********* RMTHOST targethost, MGRPORT 7809 RMTTASK REPLICAT, GROUP INITREP TABLE SALES.ACCOUNT; Example Replicat parameter file REPLICAT INITREP USERID tgtuser, PASSWORD ********* ASSUMETARGETDEFS MAP SALES.ACCOUNT, TABLE SALES.ACCOUNT; Direct Load Execution To execute the Extract and Replicat process, first define the groups within GGSCI using the SOURCEISTABLE or SPECIALRUN arguments GGSCI> ADD EXTRACT INITEXT, SOURCEISTABLE GGSCI> ADD REPLICAT INITREP, SPECIALRUN GGSCI> START EXTRACT INITEXT

Initial Data Load

Direct Bulk Load Example parameter file Example Extract parameter file EXTRACT INITEXT USERID etluser, PASSWORD ********* RMTHOST targethost, MGRPORT 7809 RMTTASK REPLICAT, GROUP INITREP TABLE SALES.ACCOUNT; Example Replicat parameter file REPLICAT INITREP USERID tgtuser, PASSWORD ********* BULKLOAD ASSUMETARGETDEFS MAP SALES.ACCOUNT, TABLE SALES.ACCOUNT; Direct Bulk Load Execution To execute the Extract and Replicat process, first define the groups within GGSCI using the SOURCEISTABLE or SPECIALRUN arguments GGSCI> ADD EXTRACT INITEXT, SOURCEISTABLE GGSCI> ADD REPLICAT INITREP, SPECIALRUN GGSCI> START EXTRACT INITEXT

Change Capture and Delivery

Extract Extract can be configured to capture changed data from Oracle online redo and archive logs DB2 logs MS SQL logs GoldenGate log table produced by triggers Extract can distribute data to other remote systems Directly from GoldenGate trails Extract can be configured for initial data load Directly from source tables Extract Configuration To configure Extract to capture changes from transaction logs Add initial Extract checkpoint into the transaction log (use GGSCI ADD EXTRACT command) Configure Extract parameter file (use GGSCI EDIT PARAMS command) Define GoldenGate trails (use GGSCI ADD RMTTRAIL or ADD EXTTRAIL commands) Start Extract (use GGSCI START EXTRACT command) Example Oracle: GGSCI> EDIT PARAMS ODS EXTRACT ODS USERID etluser, PASSWORD *********

Change Capture and Delivery

RMTHOST targethost, MGRPORT 7809 RMTTRAIL /ggs/diirdat/rt TABLE SALES.ORDERS; TABLE SALES.INVENTORY; GGSCI> ADD EXTRACT ODS, TRANLOG, BEGIN NOW GGSCI> ADD RMTTRAIL /ggs/diirdat/rt, EXTRACT ODS GGSCI> START EXTRACT ODS Example DB2 and SQL SERVER: GGSCI> EDIT PARAMS ODS EXTRACT ODS SOURCEDB dsn, USERID etluser, PASSWORD ********* RMTHOST targethost, MGRPORT 7809 RMTTRAIL /ggs/diirdat/rt TABLE SALES.ORDERS; TABLE SALES.INVENTORY; GGSCI> ADD EXTRACT ODS, TRANLOG, BEGIN NOW GGSCI> ADD RMTTRAIL /ggs/diirdat/rt, EXTRACT ODS GGSCI> START EXTRACT ODS

Change Capture and Delivery

GoldenGate Trails: As Extract captures data, it stores the captured data and a description of that data, into files known as GoldenGate trails. Trail files are created as needed and are aged automatically to allow processing to continue without interruption for file maintenance. A trail can exist on the source or target system, depending on how you configure GoldenGate. On the local system it is known as an extract trail (or local trail). On a remote system it is known as a remote trail. Only one Extract process can write to a trail. GoldenGate writes data to the trail in universal data format, a proprietary format which allows it to be exchanged rapidly and accurately among heterogeneous databases. The records in GoldenGate trails contain header and data areas. The header contains information about the transaction environment, and the data area contains the actual data values that were extracted. All file names in a trail begin with the same two characters, which you assign when you create the trail. As files are created, each name is appended with a unique six-digit serial (sequence) number from 000000 through 999999, for example \ggs\dirdat\tr000001. By default, trails are stored in the dirdat sub-directory of the GoldenGate directory Replicat

Change Capture and Delivery

Replicat - Configuration To configure Replicat to apply changes from GoldenGate trails to the target database: Create checkpoint table (use GGSCIT ADD CEHECKPOINTABLE command) Setup parameter file for Replicat (use GGSCI EDIT PARAMS command) Add initial Replicat checkpoint into GoldenGate trails (use GGSCI ADD REPLICAT command) Start the Replicat process (use GGSCI START REPLICAT command) Example: GGSCI> DBLOGIN USERID etluser PASSWORD pw GGSCI> ADD CHECKPOINTTABLE ggs.checkpt GGSCI> EDIT PARAMS REPORD REPLICAT REPORD TARGETDB dsn USERID etluser, PASSWORD ********* (For Oracle) USERID etluser, PASSWORD ********* (For DB2 or SQL SERVER) ASSUMETAREGTDFS DISCARDFILE /ggs/dirrpt/REPORT.dsc, APPEND MAP SALES.ORDERS, TARGET SALES.ORDERS; MAP SALES.INVENTORY, TARGET SALES.INVENTORY; GGSCI> ADD REPLICAT REPORD, EXTRAIL /ggs/dirdat/rt GGSCI> START REPLICAT REPORD

Conclusion

Conclusion
Other Information:
There are several other commands, parameters, functions, etc used during the Manager, Extract, Replicat Processes which we are not going in detail here. Pls refer the GGS_WinUnix_Reference_9.5.pdf & GGS_WinUnix_QuickRef_9.5.pdf documents comes with the GoldenGate software.

References:
GoldenGate website GoldenGate Fundamentals Student Guide study material GoldenGate reference manuals