ETL Testing / Data Warehouse Testing Tips, Techniques, Process and Challenges

Today let me take a moment and explain my testing fraternity about one of the much in demand and upcoming skills for my tester friends i.e. ETL testing (Extract, Transform, and Load). This article will present you with a complete idea about ETL testing and what we do to test ETL process. It has been obser ed that Independent !erification and !alidation is gaining huge market potential and many companies are now seeing this as prospecti e business gain. "ustomers ha e been offered different range of products in terms of ser ice offerings, distributed in many areas based on technology, process and solutions. ETL or data warehouse is one of the offerings which are de eloping rapidly and successfully. Why do organizations need Data Warehouse? #rgani$ations with organi$ed IT practices are looking forward to create a next le el of technology transformation. They are now trying to make themsel es much more operational with easy%to%interoperate data. &a ing said that data is most important part of any organi$ation, it may be e eryday data or historical data. 'ata is backbone of any report and reports are the baseline on which all the ital management decisions are taken. (ost of the companies are taking a step forward for constructing their data warehouse to store and monitor real time data as well as historical data. "rafting an efficient data warehouse is not an easy )ob. (any organi$ations ha e distributed departments with different applications running on distributed technology. ETL tool is employed in order to make a flawless integration between different data sources from different departments. ETL tool will work as an integrator, extracting data from different sources* transforming it in preferred format based on the business transformation rules and loading it in cohesi e '+ known are 'ata ,arehouse. ,ell planned, well defined and effecti e testing scope guarantees smooth con ersion of the pro)ect to the production. - business gains the real buoyancy once the ETL processes are erified and alidated by independent group of experts to make sure that data warehouse is concrete and robust. ETL or Data warehouse testing is categorized into our di erent engage!ents irrespecti e of technology or ETL tools used.

"ew Data Warehouse Testing / 0ew ', is built and erified from scratch. 'ata input is taken from customer re1uirements and different data sources and new data warehouse is build and erified with the help of ETL tools. #igration Testing / In this type of pro)ect customer will ha e an existing ', and ETL performing the )ob but they are looking to bag new tool in order to impro e efficiency. Change $equest / In this type of pro)ect new data is added from different sources to an existing ',. -lso, there might be a condition where customer needs to change their existing business rule or they might integrate the new rule. $eport Testing / 2eport are the end result of any 'ata ,arehouse and the basic propose for which ', is build. 2eport must be tested by alidating layout, data in the report and calculation.

ETL Testing Techniques:

3) !erify that data is transformed correctly according to arious business re1uirements and rules. 4) (ake sure that all pro)ected data is loaded into the data warehouse without any data loss and truncation. 5) (ake sure that ETL application appropriately re)ects, replaces with default alues and reports in alid data. 6) (ake sure that data is loaded in data warehouse within prescribed and expected time frames to confirm impro ed performance and scalability.

-part from these 6 main ETL testing methods other testing methods like integration testing and user acceptance testing is also carried out to make sure e erything is smooth and reliable.

ETL Testing Process:

7imilar to any other testing that lies under Independent !erification and !alidation, ETL also go through the same phase.

+usiness and re1uirement understanding !alidating Test Estimation Test planning based on the inputs from test estimation and business re1uirement 'esigning test cases and test scenarios from all the a ailable inputs #nce all the test cases are ready and are appro ed, testing team proceed to perform pre%execution check and test data preparation for testing Lastly execution is performed till exit criteria are met 8pon successful completion summary report is prepared and closure process is done.

It is necessary to define test strategy which should be mutually accepted by stakeholders before starting actual testing. - well defined test strategy will make sure that correct approach has been followed meeting the testing aspiration. ETL testing might re1uire writing 79L statements extensi ely by testing team or may be tailoring the 79L pro ided by de elopment team. In any case testing team must be aware of the results they are trying to get using those 79L statements. Di erence %etween Data%ase and Data Warehouse Testing There is a popular misunderstanding that database testing and data warehouse is similar while the fact is that both hold different direction in testing.

'atabase testing is done using smaller scale of data normally with #LT: (#nline transaction processing) type of databases while data warehouse testing is done with large olume with data in ol ing #L-: (online analytical processing) databases. In database testing normally data is consistently in)ected from uniform sources while in data warehouse testing most of the data comes from different kind of data sources which are se1uentially inconsistent. ,e generally perform only "28' ("reate, read, update and delete) operation in database testing while in data warehouse testing we use read%only (7elect) operation. 0ormali$ed databases are used in '+ testing while demorali$ed '+ is used in data warehouse testing.

There are number of uni ersal erifications that ha e to be carried out for any kind of data warehouse testing. +elow is the list of ob)ects that are treated as essential for alidation in ETL testing. % !erify that data transformation from source to destination works as expected % !erify that expected data is added in target system % !erify that all '+ fields and field data is loaded without any truncation % !erify data checksum for record count match % !erify that for re)ected data proper error logs are generated with all details % !erify 08LL alue fields % !erify that duplicate data is not loaded % !erify data integrity

ETL Testing Challenges:

ETL testing is 1uite different from con entional testing. There are many challenges we faced while performing data warehouse testing. &ere is the list of few ETL testing challenges I experienced on my pro)ect. % Incompatible and duplicate data. % Loss of data during ETL process. % 8na ailability of inclusi e test bed. % Testers ha e no pri ileges to execute ETL )obs by their own. % !olume and complexity of data is ery huge. % ;ault in business process and procedures. % Trouble ac1uiring and building test data. % (issing business flow information. 'ata is important for businesses to make the critical business decisions. ETL testing plays a significant role alidating and ensuring that the business information is exact, consistent and reliable. -lso, it minimi$es ha$ard of data loss in production. &ope these tips will help ensure your ETL process is accurate and the data warehouse build by this is a competiti e ad antage for your business.