DATA MINING

1. DATA MINING
Introduction The past two decades has seen a dramatic increase in the amount of information or data being stored in electronic format. This accumulation of data has taken place at an explosive rate. It has been estimated that the amount of information in the world doubles every 20 months and the size and number of databases are increasing even faster. The increase in use of electronic data gathering devices such as point-of-sale or remote sensing devices has contributed to this explosion of available data. The problem of effectively utilizing these massive volumes of data is becoming a major problem for all enterprises. Database Management systems gave access to the data stored but this was only a small part of what could be gained from the data. Traditional on-line transaction processing systems, OLTPs, are good at putting data into databases quickly, safely and efficiently but are not good at delivering meaningful analysis in return. Analyzing data can provide further knowledge about a business by going beyond the data explicitly stored to derive knowledge about the business. This is where Data Mining has obvious benefits for any enterprise. What is Data Mining? Definition Researchers William J Frawley, Gregory Piatetsky-Shapiro and Christopher J Matheus have defined Data Mining as: Data mining is the search for relationships and global patterns that exist in large databases but are `hidden' among the vast amount of data, such as a relationship between patient data and their medical diagnosis. These relationships represent valuable knowledge about the database and the objects in the database and, if the database is a faithful mirror, of the real world registered by the database. Data mining can be used in the areas such as decision support, prediction, forecasting and estimation. The data is often voluminous, but as it stands of low

value as no direct use can be made of it; it is the hidden information in the data that is useful Explanation Data mining, the extraction of hidden predictive information from large databases, is a powerful new technology with great potential to help companies focus on the most important information in their data warehouses. Data mining tools predict future trends and behaviours, allowing business to make proactive knowledge driven decisions. The automated, prospective analysis offered by data mining move beyond the analysis of past events provided by retrospective tools typical of decision support systems. Data mining tools can answer business questions that traditionally were too time consuming to resolve. The data mining process consists of three basic stages: exploration, model building and pattern definition. Fig. 1.1 shows a simple data mining structure.

Condition al Logic Discovery Affinities & Trends & Variations Outcome Prediction Forecastin Deviation Detection Link Analysis

Data

Predictive Modeling

Forensic Analysis Fig. 1.1 Data Mining Structure

Basically data mining is concerned with the analysis of data and the use of software techniques for finding patterns and regularities in sets of data. It is the computer

which is responsible for finding the patterns by identifying the underlying rules and features in the data. Data mining analysis makes use of as much of the collected data as possible to arrive at reliable conclusions and decisions. The analysis process starts with a set of data, uses a methodology to develop an optimal representation of the structure of the data during which time knowledge is acquired. Once knowledge has been acquired this can be extended to larger sets of data working on the assumption that the larger data set has a structure similar to the sample data. Again this is analogous to a mining operation where large amounts of low grade materials are sifted through in order to find something of value. Example A home finance loan actually has an average life span of only 7 to 10 years, due to prepayment. Prepayment means, the loan is paid off early, rather than at the end of, say 25 years. People prepay loans when they refinance or when they sell their home. The financial return that a home-finance derives from a loan depends on its life span. Therefore it is necessary for the financial institutions to be able to predict the life spans of their loans. Rule discovery techniques are used to accurately predict the aggregate number of loan payments in a given quarter (or in a year), as a function of prevailing interest rates, borrower characteristics, and account data. This information can be used to finetune loan parameters such as interest rates, points and fees, in order to maximize profits. Knowledge Discovery in Database (KDD) KDD and Data Mining Knowledge Discovery in Database (KDD) was formalized in 1989, with reference to the general concept of being broad and high level in pursuit of seeking knowledge from data. The term data mining was then coined; this high-level application technique is used to present and analyze data for decision-makers. Data mining is only one of the many steps involved in knowledge discovery in databases. The KDD process tends to be highly iterative and interactive. Data mining analysis tends to work up from the data and the best techniques are developed with an

orientation towards large volumes of data, making use of as much data as possible to arrive at reliable conclusions and decisions. The analysis process starts with a set of data, and uses a methodology to develop an optimal representation of the structure of data, during which knowledge is acquired. Once knowledge is acquired, this can be extended to large sets of data on the assumption that the large data set has a structure similar to the simple data set. Fayyad distinguishes between KDD and data mining by giving the following definitions: Knowledge discovery in databases is the process of identifying a valid, potentially useful and ultimately understandable structure in data. Data mining is a step in the KDD process concerned with the algorithmic means by which patterns or structures are enumerated from the data under acceptable computational efficiency limits. The structures that are the outcome of the data mining process must meet certain conditions so that these can be considered as knowledge. These conditions are: validity, understandability, utility, novelty and interestingness.

Other Related Areas Data Mining has drawn on a number of other fields, some of which are listed below. Statistics Statistics is a theory-rich approach for data analysis, which generates results that can be overwhelming and difficult to interpret. Not withstanding this, statistics is one of the foundations on which data mining technology is built. Statistical analysis systems are used by analysts to detect unusual patterns and explain patterns using statistical models. Statistics have an important role to play and data mining will not replace such analyses, but rather statistics can act upon more directed analyses based on the results of data mining. Machine Learning

Machine learning is the automation of a learning process and learning is tantamount to the construction of rules based on observations. This is a broad field, which includes not only learning from examples, but also reinforcement learning, learning with teacher, etc. A learning algorithm takes the data set and its accompanying information as input and returns a statement e.g. a concept representing the results of learning as output. Inductive Learning Induction is the inference of information from data and inductive learning is the model building process where the environment i.e. database is analyzed with a view to finding patterns. Similar objects are grouped in classes and rules formulated whereby it is possible to predict the class of unseen objects. This process of classification identifies classes such that each class has a unique pattern of values, which forms the class description. The nature of the environment is dynamic hence the model must be adaptive i.e. should be able learn. Inductive learning where the system infers knowledge itself from observing its environment has two main strategies: Supervised Learning and Unsupervised Learning. Supervised Learning This is learning from examples where a teacher helps the system construct a model by defining classes and supplying examples of each class. Unsupervised Learning This is learning from observation and discovery. Mathematical Programming Most of the major data mining tasks can be equivalently formulated as problems in mathematical programming for which efficient algorithms are available. It provides a new insight into the problems of data mining. Data Mining Techniques Researchers identify two fundamental goals of data mining: prediction and description. Prediction makes use of existing variables in the database in order to predict unknown or future values of interest, and description focuses on finding patterns describing the data and the subsequent presentation for user interpretation. The relative emphasis of both, prediction and description differ with respect to the underlying application and technique.

There are several data mining techniques fulfilling these objectives. Some of these are associations, classifications, sequential patterns and clustering. Another approach of the study of data mining techniques is to classify the techniques as: user-guided or verification-driven data mining and, discovery-driven or automatic discovery of rules. Most of the techniques of data mining have elements of both the models. Data Mining Models Verification Model: The verification model takes an hypothesis from the user and tests the validity of it against the data. The emphasis is with the user who is responsible for formulating the hypothesis and issuing the query on the data to affirm or negate the hypothesis. In a marketing division, for example, with a limited budget for a mailing campaign to launch a new product it is important to identify the section of the population most likely to buy the new product. The user formulates a hypothesis to identify potential customers and the characteristics they share. Historical data about customer purchase and demographic information can then be queried to reveal comparable purchases and the characteristics shared by those purchasers. The whole operation can be repeated by successive refinements of hypothesis until the required limit is reached. Discovery Model: The discovery model differs in its emphasis in that it is the system automatically discovering important information hidden in the data. The data is sifted in search of frequently occurring patterns, trends and generalizations about the data without intervention or guidance from the user. The discovery or data mining tools aim to reveal a large number of facts about the data in as short a time as possible. An example of such a model is a supermarket database, which is mined to discover the particular groups of customers to target for a mailing campaign. The data is searched with no hypothesis in mind other than for the system to group the customers according to the common characteristics found. Data Mining Methods Various data mining methods are: Neural Networks

Genetic Algorithms Rough Sets Techniques Support Vector Machines Cluster Analysis Induction OLAP Data Visualization