Professional Documents
Culture Documents
DATA MINING:-
Data mining is the process of extracting the useful information, which is
stored in the large database.
It is a powerful tool, which is useful for organizations to retrieve the useful
information from available data warehouses.
Data mining can be applied to relational databases, object-oriented databases,
data warehouses, structured-unstructured databases, etc.
Data mining is used in numerous areas like banking, insurance companies,
pharmaceutical companies etc.
Fraud Detection
Apart from these, data mining can also be used in the areas of
production control, customer retention, science exploration, sports,
astrology, and Internet Web Surf-Aid.
Market Analysis and Management
Listed below are the various fields of market where data mining is used
−
Customer Profiling − Data mining helps determine what kind of people buy what
kind of products.
Target Marketing − Data mining helps to find clusters of model customers who
share the same characteristics such as interests, spending habits, income, etc.
Fraud Detection
Data mining is also used in the fields of credit card services and
telecommunication to detect frauds. In fraud telephone calls, it helps to
find the destination of the call, duration of the call, time of the day or
week, etc. It also analyzes the patterns that deviate from expected
norms.
Steps 1 through 4 are different forms of data preprocessing, where data are
prepared for mining. The data mining step may interact with the user or a knowledge
base. The interesting patterns are presented to the user and may be stored as new
knowledge in the knowledge base.
KDD vs Datamining
KDD Data Mining
It is a field of computer science, that helps to extract Data mining is one of the important steps in the
useful and previously undiscovered knowledge from KDD process. It includes suitable algorithm based on
the large database by using various tools and the objective of the KDD process to identify the
theories. patterns from the database.
OLAP
Online analytical processing operations make use of background knowledge
regarding the domain of the data being studied to allow the presentation of data at
different levels of abstraction. Such operations accommodate different user
viewpoints. Examples of OLAP operations include drill-down and roll-up, which
allow the user to view the data at differing degrees of summarization.
OLAP (Online Analytical Processing) provides the support for the multidimensional
view of the data.
It provides easy and efficient access to the various views of information to the users.
The complex queries are also processed by using OLAP. It is easy to analyze information
by processing multidimensional views of the data.
The data warehouse is used to analyze the information, where the ample amount of
historical data is stored.
OLAP Operations
OLAP techniques are applied to retrieve the information from the data warehouse in the
form of OLAP multidimensional databases.
Multidimensional data mining (also called exploratory multidimensional data mining) performs
data mining in multidimensional space in an OLAP style. That is, it allows the exploration of multiple
combinations of dimensions at varying levels of granularity in data mining,
and thus has greater potential for discovering interesting patterns representing knowledge.
1. Roll up or consolidation
Descriptive Function
The descriptive function deals with the general properties of data in the
database. Here is the list of descriptive functions −
Class/Concept Description
Mining of Associations
Mining of Correlations
Mining of Clusters
I. Class/Concept Description
Class/Concept refers to the data to be associated with the classes or
concepts. For example, in a company, the classes of items for sales
include computer and printers, and concepts of customers include big
spenders and budget spenders. Such descriptions of a class or a concept
are called class/concept descriptions. These descriptions can be derived
by the following two ways −
Data Characterization −is a summarization of the general characteristics
or features of a target class of data. The data corresponding to the user-
specified class are typically collected by a query.
Simple data summaries based on statistical measures. The data cube-based
OLAP roll-up operation (Section 1.3.2) can be used to perform user-controlled
data summarization along a specified dimension An attribute-oriented
induction technique can be used to perform data generalization and
characterization without step-by-step user interaction.
The output of data characterization can be presented in various forms.
Examples include pie charts, bar charts, curves,multidimensional data
cubes, and multidimensional tables, including crosstabs. The resulting
descriptions can also be presented as generalized relations or in rule form
(called characteristic rules).
V. Mining of Clusters
Cluster refers to a group of similar kind of objects. Cluster analysis refers
to forming group of objects that are very similar to each other but are
highly different from the objects in other clusters.
PREDICTIVE FUNCTIONS
Classification is the process of finding a model (or function) that describes and
distinguishes data classes or concepts. The model are derived based on the
analysis of a set of training data (i.e., data objects for which the class labels are
known). The model is used to predict the class label of objects for which the the
class label is unknown.
The derived model can be presented in the following forms −
Decision Trees
Mathematical Formulae
Neural Networks
REGRESSION
Whereas classification predicts categorical (discrete, unordered) labels, regression
models continuous-valued functions. That is, regression is used to predict missing or
unavailable numerical data values rather than (discrete) class labels. The term
prediction refers to both numeric prediction and class label prediction.
Regression analysis is a statistical methodology that is most often used for numeric
prediction, although other methods exist as well. Regression also encompasses the
identification of distribution trends based on the available data.
Classification and regression may need to be preceded by relevance analysis,
which attempts to identify attributes that are significantly relevant to the classification
and regression process. Such attributes will be selected for the classification and
regression process. Other attributes, which are irrelevant, can then be excluded from
consideration.
Database Attributes
Characterization
Discrimination
Classification
Prediction
Clustering
Outlier Analysis
Evolution Analysis
Background knowledge
The background knowledge allows data to be mined at multiple levels of
abstraction. For example, the Concept hierarchies are one of the
background knowledge that allows data to be mined at multiple levels of
abstraction.
Tables
Charts
Graphs
Decision Trees
Cubes
Performance Issues
Data mining query languages and ad hoc data mining − Data Mining Query
language that allows the user to describe ad hoc mining tasks, should be integrated
with a data warehouse query language and optimized for efficient and flexible data
mining.
Presentation and visualization of data mining results − Once the patterns are
discovered it needs to be expressed in high level languages, and visual
representations. These representations should be easily understandable.
Handling noisy or incomplete data − The data cleaning methods are required to
handle the noise and incomplete objects while mining the data regularities. If the
data cleaning methods are not there then the accuracy of the discovered patterns
will be poor.
Performance Issues
There can be performance-related issues such as follows −
Efficiency and scalability of data mining algorithms− In order to effectively
extract the information from huge amount of data in databases, data mining
algorithm must be efficient and scalable.
DATA WAREHOUSE
A data warehouse is a repository of information collected from multiple sources,
stored under a unified schema, and usually residing at a single site. Data
warehouses are constructed via a process of data cleaning, data integration, data
transformation, data loading, and periodic data refreshing.
Time Variant − The data collected in a data warehouse is identified with a particular
time period. The data in a data warehouse provides information from a historical
point of view.
Non-volatile − Nonvolatile means the previous data is not removed when new data
is added to it. The data warehouse is kept separate from the operational database
therefore frequent changes in operational database is not reflected in the data
warehouse.
Data Warehousing
Data warehousing is the process of constructing and using the data
warehouse. A data warehouse is constructed by integrating the data from
multiple heterogeneous sources. It supports analytical reporting,
structured and/or ad hoc queries, and decision making.
Query-Driven Approach
This is the traditional approach to integrate heterogeneous databases.
This approach is used to build wrappers and integrators on top of
multiple heterogeneous databases. These integrators are also known as
mediators.
Now these queries are mapped and sent to the local query processor.
The results from heterogeneous sites are integrated into a global answer set.
Disadvantages
This approach has the following disadvantages −
The Query Driven Approach needs complex integration and filtering processes.
Update-Driven Approach
Today's data warehouse systems follow update-driven approach rather
than the traditional approach discussed earlier. In the update-driven
approach, the information from multiple heterogeneous sources is
integrated in advance and stored in a warehouse. This information is
available for direct querying and analysis.
Advantages
This approach has the following advantages −
This approach provides high performance.
Online selection of data mining functions − Integrating OLAP with multiple data
mining functions and online analytical mining provide users with the flexibility to
select desired data mining functions and swap data mining tasks dynamically.
1. The bottom tier is a warehouse database server that is almost always a relational database
system. Back-end tools and utilities are used to feed data into the bottom tier from operational
databases or other external sources (e.g., customer profile information provided by external
consultants). These tools and utilities performdata extraction, cleaning, and transformation (e.g., to
merge similar data from different sources into a unified format), as well as load and refresh functions
to update the data warehouse.
The data are extracted using application program interfaces known as gateways. A gateway is
supported by the underlying DBMS and allows client programs to generate SQL code to be executed
at a server.
This tier also contains a metadata repository, which stores information about the data warehouse and
its contents.
2. The middle tier is an OLAP server that is typically implemented using either (1) a relational
OLAP(ROLAP) model (i.e., an extended relational DBMS that maps operations on multidimensional
data to standard relational operations); or (2) a multidimensional OLAP (MOLAP) model (i.e., a
special-purpose server that directly implements multidimensional data and operations).
3. The top tier is a front-end client layer, which contains query and reporting tools, analysis tools,
and/or data mining tools (e.g., trend analysis, prediction, and so on).
Data cube
A data cube allows data to be modeled and viewed in multiple dimensions. It is defined by
dimensions and facts.
Star schema: The most common modeling paradigm is the star schema, in which
the data warehouse contains (1) a large central table (fact table) containing the bulk
of the data, with no redundancy, and (2) a set of smaller attendant tables (dimension
tables), one for each dimension. The schema graph resembles a starburst, with the
dimension tables displayed in a radial pattern around the central fact table.
Snowflake schema: The snowflake schema is a variant of the star schema model,
where some dimension tables are normalized, thereby further splitting the data into
additional tables. The resulting schema graph forms a shape similar to a snowflake.
The major difference between the snowflake and star schema models is that the
dimension tables of the snowflake model may be kept in normalized form to reduce
redundancies. Such a table is easy to maintain and saves storage space. However,
this space savings is negligible in comparison to the typical magnitude of the fact
table. Furthermore, the snowflake structure can reduce the effectiveness of browsing,
since more joins will be needed to execute a query. Consequently, the system
performance may be adversely impacted.Hence, although the snowflake schema
reduces redundancy, it is not as popular as the star schema in data warehouse
design.
MEASURES
A multidimensional point in the data cube space can be defined by a set of dimension–value
pairs. A data cube measure is a numeric function that can be evaluated at each point in the
data cube space. A measure value is computed for a given point by aggregating the data
corresponding to the respective dimension–value pairs defining the given point. We will look
at concrete examples of this shortly.
Drill-down: Drill-down is the reverse of roll-up. It navigates from less detailed data to more
detailed data. Drill-down can be realized by either stepping down a concept hierarchy for a
dimension or introducing additional dimensions. Because a drill-down adds more detail to the
given data, it can also be performed by adding new dimensions to a cube.
Slice and dice: The slice operation performs a selection on one dimension of the given cube,
resulting in a subcube. The dice operation defines a subcube by performing a selection on
two or more dimensions.
Pivot (rotate): Pivot (also called rotate) is a visualization operation that rotates the data axes
in view to provide an alternative data presentation.
Other OLAP operations: Some OLAP systems offer additional drilling operations.
drill-across executes queries involving (i.e., across) more than one fact table.
The drill-through operation uses relational SQL facilities to drill through the bottom
level of a data cube down to its back-end relational tables.
Other OLAP operations may include ranking the top N or bottom N items in lists, as
well as computing moving averages, growth rates, interests, internal return rates,
depreciation, currency conversions, and statistical functions.