After reading this article you will learn about:- 1. Meaning of Data Mining 2. Concept of Data Mining 3. Foundations 4. Scope 5. Working 6. Architecture 7. Profitable Applications.
Meaning of Data Mining:
In the CRM context, data mining can be defined as follows:
Data mining is the application of descriptive and predictive analytics to support the marketing, sales and service functions. Although data mining can be performed on operational databases, it is more commonly applied to the more stable datasets held in data marts or warehouses.
Higher processing speeds, reduced storage costs and better software packages have made data mining more attractive and economical. Data mining can provide answers to questions that are important for both strategic and operational CRM purposes.
For example:
1. How can our market and customer base be segmented?
2. Which customers are most valuable?
3. Which customers offer most potential for the future?
4. What types of customers are buying our products? Or not buying?
5. Are there any patterns of purchasing behaviour in our customer base?
6. Should we charge the same price to all these segments?
7. What is the profile of customers who default on payment?
8. What are the costs of customer acquisition?
9. What sorts of customer should be targeted for acquisition?
10. What offers should be made to specific customer groups to increase their value?
11. Which customers should be targeted for customer retention efforts?
12. Which retention tactics work well?
Data mining helps CRM in a number of ways. It can find associations between data. For example, the data may reveal that customers who buy low fat desserts are also big buyers of herbal health and beauty aids, or that consumers of wine enjoy live theatre productions.
One analyst at Wal-Mart, the American retailer, noted a correlation between diaper sales and beer sales, which was particularly strong on Fridays. On investigating further he found that fathers were buying the diapers and picking up a six-pack at the same time. The company responded to this information by locating these items closer to each other. Sales of both rose strongly.
Sequential patterns often emerge from data mining. Data miners look for ‘ if … then ‘ rules in customer behaviour. For example, they might find a rule such as ‘ If a customer buys walking shoes in November, then there is a 40 per cent probability that they will buy rainwear within the next six months ‘, or ‘ If a customer calls a contact centre to request information about interest rates, then there is a 50 per cent probability the customer will churn in the next three months ‘. Rules such as these enable CRM users to implement timely tactics. In the first instance, there is an opportunity for cross-selling. Secondly, there may be an opportunity to save the customer.
Data-mining also works by classifying. Customers can be classified into mutually exclusive groups. For example, you might be able to segment your existing customers into groups according to the value they produce for your company.
You can then profile each group. When you identify a potential new customer you can judge which group the prospect most resembles. That will give you an idea of the prospect’s potential value. You could also classify customers into quintiles or declines in terms of important transactional information such as the recency, frequency and monetary value of the purchases they have made. This is called RFM analysis.
Another approach in data mining is clustering . CRM practitioners attempt to cluster customers into groups. The general objective of clustering is to minimize the differences between members of a cluster while also maximizing the differences between clusters.
Clustering techniques work by using a defined range of variables to perform the clustering procedure. You might, for example, use all available transaction data to generate customer segments. There are a number of techniques, such as cluster analysis, which find the hidden clusters. Once statistical clusters have been formed they need to be interpreted.
Lifestyle market segments are outputs of cluster analysis on large sets of data. Cluster labels such as ‘ Young working class families ‘ or ‘ Wealthy suburbanites ‘ are often used to capture the essence of the cluster. Finally, data mining can contribute to CRM by making predictions .CRM practitioners might use historic purchasing behaviour to predict future purchasing behaviour and customer lifetime value.
These five major approaches to data mining can be used in various sequences. For example, you could use clustering to create customer segments, then within segments use transactional data to predict future purchasing and customer lifetime value. According to Gartner Inc., market leaders SAS and SPSS offer broad data mining solutions that meet most market needs. 15 There are many other vendors.
Successful vendors of CRM analytics provide the following:
i. packaged applications to support common CRM decisions such across-sell and customer churn prediction.
ii. a user interface suitable for business users.
iii. the capability to access data from various sources including data ware houses, data marts, call centres, e-commerce or web-tracking systems, as well as third party data sources.
iv. robust data mining statistical tools such as cluster analysis, decision trees and neural networks that can provide reliable insights into different types and volumes of data.
v. reporting tools that make the results of analysis available to decision makers such as campaign managers and call centre agents.
Concept of Data Mining:
Data mining, the extraction of hidden predictive information from large databases, is a powerful new technology with great potential to help companies focus on the most important information in their data warehouses.
Data mining tools predict future trends and behaviors, allowing businesses to make proactive, knowledge-driven decisions. The automated, prospective analyses offered by data mining move beyond the analyses of past events provided by retrospective tools typical of decision support systems.
Data mining tools can answer business questions that traditionally were too time consuming to resolve. They scour databases for hidden patterns, finding predictive information that experts may miss because it lies outside their expectations.
Most companies already collect and refine massive quantities of data. Data mining techniques can be implemented rapidly on existing software and hardware platforms to enhance the value of existing information resources, and can be integrated with new products and systems as they are brought on-line. When implemented on high performance client/server or parallel processing computers, data mining tools can analyze massive databases to deliver answers to questions such as, “Which clients are most likely to respond to my next promotional mailing, and why?”
The Foundations of Data Mining:
Data mining techniques are the result of a long process of research and product development. This evolution began when business data was first stored on computers, continued with improvements in data access, and more recently, generated technologies that allow users to navigate through their data in real time.
Data mining takes this evolutionary process beyond retrospective data access and navigation to prospective and proactive information delivery.
Data mining is ready for application in the business community because it is supported by three technologies that are now sufficiently mature:
i. Massive data collection
ii. Powerful multiprocessor computers
iii. Data mining algorithms
Commercial databases are growing at unprecedented rates. A recent META Group survey of data warehouse projects found that 19% of respondents are beyond the 50 gigabyte level, while 59% expect to be there by second quarter of 1996.
In some industries, such as retail, these numbers can be much larger. The accompanying need for improved computational engines can now be met in a cost-effective manner with parallel multiprocessor computer technology Data mining algorithms embody techniques that have existed for at least 10 years, but have only recently been implemented as mature, reliable, understandable tools that consistently outperform older statistical methods.
In the evolution from business data to business information, each new step has built upon the previous one. For example, dynamic data access is critical for drill-through in data navigation applications, and the ability to store large databases is critical to data mining. From the user’s point of view, the four steps listed in Table 1 were revolutionary because they allowed new business questions to be answered accurately and quickly.
The core components of data mining technology have been under development for decades, in research areas such as statistics, artificial intelligence, and machine learning. Today, the maturity of these techniques, coupled with high-performance relational database engines and broad data integration efforts, make these technologies practical for current data warehouse environments.
The Scope of Data Mining:
Data mining derives its name from the similarities between searching for valuable business information in a large database — for example, finding linked products in gigabytes of store scanner data — and mining a mountain for a vein of valuable ore. Both processes require either sifting through an immense amount of material, or intelligently probing it to find exactly where the value resides.
Given databases of sufficient size and quality, data mining technology can generate new business opportunities by providing these capabilities:
i. Automated prediction of trends and behaviors. Data mining automates the process of finding predictive information in large databases. Questions that traditionally required extensive hands-on analysis can now be answered directly from the data — quickly. A typical example of a predictive problem is targeted marketing. Data mining uses data on past promotional mailings to identify the targets most likely to maximize return on investment in future mailings. Other predictive problems include forecasting bankruptcy and other forms of default, and identifying segments of a population likely to respond similarly to given events.
ii. Automated discovery of previously unknown patterns. Data mining tools sweep through databases and identify previously hidden patterns in one step. An example of pattern discovery is the analysis of retail sales data to identify seemingly unrelated products that are often purchased together. Other pattern discovery problems include detecting fraudulent credit card transactions and identifying anomalous data that could represent data entry keying errors.
Data mining techniques can yield the benefits of automation on existing software and hardware platforms, and can be implemented on new systems as existing platforms are upgraded and new products developed.
When data mining tools are implemented on high performance parallel processing systems, they can analyze massive databases in minutes. Faster processing means that users can automatically experiment with more models to understand complex data. High speed makes it practical for users to analyze huge quantities of data. Larger databases, in turn, yield improved predictions.
Databases can be larger in both depth and breadth:
i. More columns:
Analysts must often limit the number of variables they examine when doing hands-on analysis due to time constraints. Yet variables that are discarded because they seem unimportant may carry information about unknown patterns. High performance data mining allows users to explore the full depth of a database, without preselecting a subset of variables.
ii. More rows:
Larger samples yield lower estimation errors and variance, and allow users to make inferences about small but important segments of a population. A recent Gartner Group Advanced Technology Research Note listed data mining and artificial intelligence at the top of the five key technology areas that “will clearly have a major impact across a wide range of industries within the next 3 to 5 years.” Gartner also listed parallel architectures and data mining as two of the top 10 new technologies in which companies will invest during the next 5 years.
According to a recent Gartner HPC Research Note, “With the rapid advance in data capture, transmission and storage, large-systems users will increasingly need to implement new and innovative ways to mine the after-market value of their vast stores of detail data, employing MPP [massively parallel processing] systems to create new sources of business advantage (0.9 probability).”
The most commonly used techniques in data mining are:
i. Artificial neural networks:
Non-linear predictive models that learn through training and resemble biological neural networks in structure.
ii. Decision trees:
Tree-shaped structures that represent sets of decisions. These decisions generate rules for the classification of a dataset. Specific decision tree methods include Classification and Regression Trees (CART) and Chi Square Automatic Interaction Detection (CHAID).
iii. Genetic algorithms:
Optimization techniques that use processes such as genetic combination, mutation, and natural selection in a design based on the concepts of evolution.
iv. Nearest neighbor method:
A technique that classifies each record in a dataset based on a combination of the classes of the k record(s) most similar to it in a historical dataset (where k 31). Sometimes called the k-nearest neighbor technique.
v. Rule induction:
The extraction of useful if then rules from data based on statistical significance. Many of these technologies have been in use for more than a decade in specialized analysis tools that work with relatively small volumes of data. These capabilities are now evolving to integrate directly with industry-standard data warehouse and OLAP platforms. The appendix to this white paper provides a glossary of data mining terms.
Working of Data Mining:
How exactly is data mining able to tell you important things that you didn’t know or what is going to happen next? The technique that is used to perform these feats in data mining is called modeling. Modeling is simply the act of building a model in one situation where you know the answer and then applying it to another situation that you don’t.
For instance, if you were looking for a sunken Spanish galleon on the high seas the first thing you might do is to research the times when Spanish treasure had been found by others in the past.
You might note that these ships often tend to be found off the coast of Bermuda and that there are certain characteristics to the ocean currents, and certain routes that have likely been taken by the ship’s captains in that era.
You note these similarities and build a model that includes the characteristics that are common to the locations of these sunken treasures. With these models in hand you sail off looking for treasure where your model indicates it most likely might be given a similar situation in the past. Hopefully, if you’ve got a good model, you find your treasure.
This act of model building is thus something that people have been doing for a long time, certainly before the advent of computers or data mining technology. What happens on computers, however, is not much different than the way people build models.
Computers are loaded up with lots of information about a variety of situations where an answer is known and then the data mining software on the computer must run through that data and distill the characteristics of the data that should go into the model.
Once the model is built it can then be used in similar situations where you don’t know the answer. For example, say that you are the director of marketing for a telecommunications company and you’d like to acquire some new long distance phone customers.
You could just randomly go out and mail coupons to the general population – just as you could randomly sail the seas looking for sunken treasure. In neither case would you achieve the results you desired and of course you have the opportunity to do much better than random – you could use your business experience stored in your database to build a model.
As the marketing director you have access to a lot of information about all of your customers: their age, sex, credit history and long distance calling usage. The good news is that you also have a lot of information about your prospective customers: their age, sex, credit history etc.
Your problem is that you don’t know the long distance calling usage of these prospects (since they are most likely now customers of your competition). You’d like to concentrate on those prospects who have large amounts of long distance usage. You can accomplish this by building a model. Table 2 illustrates the data used for building a model for new customer prospecting in a data warehouse.
The goal in prospecting is to make some calculated guesses about the information in the lower right hand quadrant based on the model that we build going from Customer General Information to Customer Proprietary Information.
For instance, a simple model for a telecommunications company might be:
98% of my customers who make more than $60,000/year spend more than $80/month on long distance
This model could then be applied to the prospect data to try to tell something about the proprietary information that this telecommunications company does not currently have access to. With this model in hand new customers can be selectively targeted.
Test marketing is an excellent source of data for this kind of modeling. Mining the results of a test market representing a broad but relatively small sample of prospects can provide a foundation for identifying good prospects in the overall market. Table 3 shows another common scenario for building models: predict what is going to happen in the future.
If someone told you that he had a model that could predict customer usage how would you know if he really had a good model? The first thing you might try would be to ask him to apply his model to your customer base where you already knew the answer. With data mining, the best way to accomplish this is by setting aside some of your data in a vault to isolate it from the mining process. Once the mining is complete, the results can be tested against the data held in the vault to confirm the model’s validity. If the model works, its observations should hold for the vaulted data.
An Architecture for Data Mining:
To best apply these advanced techniques, they must be fully integrated with a data warehouse as well as flexible interactive business analysis tools. Many data mining tools currently operate outside of the warehouse, requiring extra steps for extracting, importing, and analyzing the data.
Furthermore, when new insights require operational implementation, integration with the warehouse simplifies the application of results from data mining. The resulting analytic data warehouse can be applied to improve business processes throughout the organization, in areas such as promotional campaign management, fraud detection, new product rollout, and so on. Figure 1 illustrates an architecture for advanced analysis in a large data warehouse. Illustrates an architecture for advanced analysis in a large data warehouse.
The ideal starting point is a data warehouse containing a combination of internal data tracking all customer contact coupled with external market data about competitor activity. Background information on potential customers also provides an excellent basis for prospecting. This warehouse can be implemented in a variety of relational database systems – Sybase, Oracle, Redbrick, and so on, and should be optimized for flexible and fast data access.
An OLAP (On-Line Analytical Processing) server enables a more sophisticated end-user business model to be applied when navigating the data warehouse. The multidimensional structures allow the user to analyze the data as they want to view their business – summarizing by product line, region, and other key perspectives of their business. The Data Mining Server must be integrated with the data warehouse and the OLAP server to embed ROI-focused business analysis directly into this infrastructure.
An advanced, process-centric metadata template defines the data mining objectives for specific business issues like campaign management, prospecting, and promotion optimization. Integration with the data warehouse enables operational decisions to be directly implemented and tracked. As the warehouse grows with new decisions and results, the organization can continually mine the best practices and apply them to future decisions.
This design represents a fundamental shift from conventional decision support systems. Rather than simply delivering data to the end user through query and reporting software, the Advanced Analysis Server applies users’ business models directly to the warehouse and returns a proactive analysis of the most relevant information. These results enhance the metadata in the OLAP Server by providing a dynamic metadata layer that represents a distilled view of the data. Reporting, visualization, and other analysis tools can then be applied to plan future actions and confirm the impact of those plans.
Profitable Applications of Data Mining:
A wide range of companies have deployed successful applications of data mining. While early adopters of this technology have tended to be in information-intensive industries such as financial services and direct mail marketing, the technology is applicable to any company looking to leverage a large data warehouse to better manage their customer relationships.
Two critical factors for success with data mining are – a large, well-integrated data warehouse and a well- defined understanding of the business process within which data mining is to be applied (such as customer prospecting, retention, campaign management, and so on).
Some successful application areas include:
a. A pharmaceutical company can analyze its recent sales force activity and their results to improve targeting of high-value physicians and determine which marketing activities will have the greatest impact in the next few months. The data needs to include competitor market activity as well as information about the local health care systems. The results can be distributed to the sales force via a wide-area network that enables the representatives to review the recommendations from the perspective of the key attributes in the decision process. The ongoing, dynamic analysis of the data warehouse allows best practices from throughout the organization to be applied in specific sales situations.
b. A credit card company can leverage its vast warehouse of customer transaction data to identify customers most likely to be interested in a new credit product. Using a small test mailing, the attributes of customers with an affinity for the product can be identified. Recent projects have indicated more than a 20-fold decrease in costs for targeted mailing campaigns over conventional approaches.
c. A diversified transportation company with a large direct sales force can apply data mining to identify the best prospects for its services. Using data mining to analyze its own customer experience, this company can build a unique segmentation identifying the attributes of high-value prospects. Applying this segmentation to a general business database such as those provided by Dun & Bradstreet can yield a prioritized list of prospects by region.
d. A large consumer package goods company can apply data mining to improve its sales process to retailers. Data from consumer panels, shipments, and competitor activity can be applied to understand the reasons for brand and store switching. Through this analysis, the manufacturer can select promotional strategies that best reach their target customer segments.
Each of these examples have a clear common ground. They leverage the knowledge about customers implicit in a data warehouse to reduce costs and improve the value of customer relationships. These organizations can now focus their efforts on the most important (profitable) customers and prospects, and design targeted marketing strategies to best reach them.
Data mining involves six common classes of tasks:
a. Anomaly detection (Outlier/change/deviation detection). The identification of unusual data records, that might be interesting or data errors and require further investigation.
b. Association rule learning (Dependency modeling). Searches for relationships between variables. For example a supermarket might gather data on customer purchasing habits. Using association rule learning, the supermarket can determine which products are frequently bought together and use this information for marketing purposes. This is sometimes referred to as market basket analysis.
c. Clustering is the task of discovering groups and structures in the data that are in some way or another “similar”, without using known structures in the data.
d. Classification is the task of generalizing known structure to apply to new data. For example, an email program might attempt to classify an email as legitimate or spam.
e. Regression attempts to find a function which models the data with the least error.
f. Summarization providing a more compact representation of the data set, including visualization and report generation.