Predicting Customer Behavior Using Statistical Techniques - Example

Name: Tutor: Title: Predicting Customer Behavior Using statistical Techniques. Course: Date: 1.0 Introduction The retail industry has experienced rapid transformation in the past decade. This dynamic and continual change in retail sales has been associated to an increase in competition, an increasing in the range of product offerings, and an increase in consumer complexity. The wide range of challenges that retailers are likely to face become increasingly pervasive as the global population of consumers steadily increases and is presented with a wide variety of choices. To be able to understand their business environment and anticipate consumer behavior, retailers are required to take an analytical, predictive and guided approach. For many years data mining had been useful only to a limited audience that had prerequisite expertise in math and statistics. With the current development in user-friendly software, simplified data mining tools are now available for ordinary retailers. Retailers can now take advantage of enhanced capabilities of data mining software to better analyze true product demand of products and optimize merchandising assortments (Berry, Michael & Gordon, 2007). This paper explores predictive analysis tools that can enable retailers to identify variables that are likely to have substantial impact on customer behavior and overall business performance. 2.0 Predictive Analysis Predictive analysis is the process by which business select, explore, analyze and model data in order to make better business decisions and achieve better business outcomes. For this process to be successful, it’s important for businesses to first establish where the required data is located, whether the data is up-to-date, and whether it is readily accessible. An important consideration in predictive analysis is the format that available data needs to be structured in order to capture business objectives in a way that reflects the actual reality of the business. Analytical models provided by data mining technologies make it possible for business to discover hidden consumer patterns that can be applied to predict future consumer developments and behavior. Predictive analysis solutions have evoked significant interest from retail organizations in the recent past, though their application at organizational level is still limited. Nonetheless, there are various data mining components that are essential in acquiring meaningful insight that is actionable. Data mining processes enable organizations to define better their business problems, explore relevant data, create predictive models, test the models, and effectively apply one or a combination of models to a target customer population to predict their behavior (Groth, 1998). Typically, companies fail to follow the complete data mining cycle, rendering the results inadequate to provide noteworthy insight. Additionally, many organizations have limited resources and information that is necessary for facilitating an enabling data mining environment, largely because a majority of organizations focus more on traditional retrospective analyses rather than contemporary predictive analysis. Modern software such as Microsoft SQL Server 2008 Data Mining add-ins provides predictive analyses that are simplified and intuitive. This makes them appealing to business analysts and key decision makers. 3.0 Data Mining Models 3.1 Defining the Business Problem A successful data mining project must start by first defining the business problem in quantitative terms. An example of an appropriate definition of a problem is: the objective of this analysis is to establish the top 30 % of our consistent customer base that is likely to churn, or, what percentage of our customer base represents the high margin segment? In the absence of a distinctively defined business problem or lack of a metric, the project may not deliver meaningful and actionable results because users will not have specific measures for the outcome. The people managing the data mining project must ensure they have adequate information about the data that they intend to put to use. Additional specifications include details of where the data will be found, who has responsibility over the data, how the data can be accessed, how much relevant data is available, whether the data satisfies the business problem, whether the data is enough to accomplish the intended business objectives, and where the data is to be processed. It is possible for sources of data to be in several locations, in multiple file formats, and to have various underlying structures. Sources of data include transactional systems, feeds available on web logs, third party data and responses from text ads. A platform such as the SQL Server 2008 provides a range of technologies that can be used to effectively automate and simplify tasks for obtaining data which adequately supports the process and addresses the business problem. For instance, the SQL Server Integration Services (SSIS) enables users to extract data from multiple sources; clean the data, convert it into the required formats, and easily load it into a readily accessible location. 3.2 Exploring the Data Exploring data involves activities that examine the health, distribution, shape, and the overall state of the data. Being able to establish the patterns and anomalies at this stage is useful in ensuring a proper comprehension of the overall business processes. Most importantly, data exploration enables analysts to identify values that may be missing, abnormal distributions, outliers, or uncommon patterns in the underlying data structure. An occurrence like where there are substantial amounts of data that is missing may imply the customer service people are inaccurately capturing customer information. Real problems may be uncovered at this stage, such as lack of proper records, or missing information on product deliveries, which points at a real problem within the business system rather than a data entry error. Analysts can use the clustering technique when exploring data. This technique exposes patterns that had hitherto been neglected or had not been detected. For instance, a retail outlet that has four segments of consumers grouped on the basis of their consumption patterns can improve customer loyalty through optimization of the merchandise assortment that is customized to meet the needs of the segments 3.3 Cleaning Data Data cleaning requires multiple repetitions in order to generate the required data set that is appropriate and aligned with business objectives. It is possible that the underlying data takes on multiple values, which are often referred to as ‘noisy’ data. It is probable that anomalous values could in actual case be valid values. Analysts must take caution when undertaking the cleaning process because exceptions can be defined by patterns within the available dataset and may not necessarily be peculiar in absolute context. 3.4 Partitioning Data The data partitioning stage entails putting in place training and testing groups for the preferred model process. The training group is a designated group or category of data that is useful in informing the model on emerging patterns and influences. The testing group is used in the assessment of the model’s validity, providing a comparison between predictive values and actual data. The training group is used by analysts to come up the first sequence of models, while the testing group is used to establish whether the generated model behaves as it is expected. This technique is sometimes referred to as ‘check and balance’. Data mining add-ins provided in Office 2010 for instance enable users to perform most tasks required in the data partitioning stage. Users are provided with multiple sampling techniques from which to choose in creating training and testing groups. Nonetheless, it is possible for predictive analysts to oversample underrepresented populations. The provided add-ins in office 2010 enables analysts to force an equal representation of several categories of data in their analysis when this is not the actual case. This ensures that the cases in the final set of data are equally represented. In the event that analysts have rare cases in their data, oversampling enables them to assess and understand behavioral patterns of the rare cases. It is important to recognize that it is not possible to undertake such processes in traditional sampling. 3.5 Generating a Model A model can be explained as an intelligent process or application that can be applied to data to detect significant patterns or trends that are likely to have substantial impact on a business’s objectives or operations. These patterns can then be used to predict probable outcomes. This empowers decision makers in a business to undertake preemptive action and positively influence future outcomes. For instance, in solving the problem of customer churn through identification of customers within a specific customer category that are likely to attrite, it is possible to apply a model that helps to proactively recognize a category of customers that is more likely to exit (Figure 1). Figure 1: Churn Influencers by Duration (2008) Number of weeks After Measuring Churn Frequency Percent 0 103 50.9 1 7 3.43 2 16 7.84 3 1 0.49 4 1 0.49 6 1 0.49 7 2 0.98 8 1 0.49 Not Significant 72 35.29 As illustrated in Figure 1, it can be established that more than half of analyses potential influencers of the churn significantly affects the churn count and this is within the same week the customers churned. 3% influenced churning a week after they churned, while 8% influenced the churn after 2 weeks. It is also depicted that a marginal percentage had influence on the churn after 38 weeks. Based on these results, retailers can either make steps to contact these customers with a promotion that will fortify the relationship, or they can choose to exclude them in future promotions to increase response rates. 3.6 Validating the Model After models have been generated, the next step is to evaluate whether they can deliver against applied data. It is possible for analysts to develop multiple models, but it may be such a daunting task to try to manage them when solving a business problem. A more practical approach would be to create multiple models, assess the performance of each of them, and then use the one that is likely to achieve the most accurate outcome. It is important to recognize that a model’s accuracy is largely depended on the data. Essentially, data mining is a progressive process that entails repeatedly measuring and comparing different models as new data is generated 3.7 Transforming the Model into Production Predictive analysis software enables statistical analysts to embed data mining into organizational processes. A good example of such a process is instantaneous detection of fraud at the very time a transaction is made in a sales outlet. By employing a data mining model, it is possible to determine a customer’s fraudulent behavior by measuring variables such as the number of transactions, total cost of transaction, and checkout duration. 4.0 Data Mining in Retail Data mining is a tool that can effectively be used to enhance and significantly expand organizations knowledge of its assets, be it customers, suppliers, employees or even how merchandise is presented in a store. Data mining can be used to solve a range of retail business challenges that range from customer segmentation, marketing campaign effectiveness, market basket determination, inventory prediction, to even demand planning. 4.1 Marketing Campaign Effectiveness As consumer shopping channels evolve and expand retailers area compelled to institute new measures that arise from these developments. With the rapid innovation in technology witnessed in present times that encompass increased reliance on the web for shopping, traditional marketing campaigns previously employed such as flyers, mailers and catalogs cannot guarantee marketers that their communication will reach the targeted audience. Important factors to consider when selecting a marketing advertisement include: what is the target audience?, what is the most effective format for the advertisement?, what is the likelihood of the recipient positively responding to the message?, how frequent can the targeted recipient be able to receive direct offers?, what is the opportunity cost in the event the customer fails to respond to the advertisement?, and lastly, what is the potential benefit in terms of value is going to be earned in the event the customer positively responds to the advertisement?. Retailers need to craft their marketing strategies based on the understanding that consumer behavior is predetermined by the available answers to these questions. This is essentially the significance of conducting a predictive analysis of the market environment before crafting and implementing marketing campaigns. Data mining helps marketers to accurately target the right customers, increase response rates for marketing campaigns, reduce related costs, and ultimately, enhance customer satisfaction and encourage repeat purchases. The worst marketers should do is to bombard customers with senseless marketing communication. With well-planned data mining and effective implementation of appropriate models, organizations are better able to target the right customer segments, and with the correct product offerings. This is based on the understanding that as consumer budgets become tighter, competition certainly increases as margins shrink. It is hence imperative that retailers become more judicious to whom, why, when and how they approach their target audiences. From the previous campaign data provided in provided in Figure 2, the retailers can be able to tell who responded and who not respond to the campaign. The retailer can use this information to create a data mining model accurately predicts the category of customers that are likely to respond to such a campaign in the future. Fig 2: Sample of campaign data (2008) By using the Data Mining Client, retailers can explore, clean and partition the data, generate a model and finally assess the model’s performance. Figure 3 provides response rate which can be expected after applying a predictive model to the marketing campaign. In this example, a predictive model can be used to achieve about 80% return in customer response rate when only 60% of the customer population is targeted. This is vital information that can significantly influence future campaigns. Figure 3: Assessing model performance (2008) 4.2 Market Segmentation By clustering, retailers can be able to segment target markets. Clustering provides a framework for categorizing unstructured data into categories with similar attributes and properties. This is typical in retail environments where customer data is used to create segments that accurately define consumers with similar consumption habits, or unique behaviors and patterns. For instance, retailers can use clustering to categorize customers as either low spend, high spend or high spend. It may also be effective in identifying purchasing patterns in stores. This would significantly influence the way retailers choose to furnish their stores with merchandise, or how marketers communicate with customers. Segmentation enables retailers to better understand consumer spending patterns, communication preferences, as well as merchandising preferences. Retailers can then effectively categorize customer segments based on these characteristics. This can also be applied when segmenting stores, vendors, or partners. A comprehensive understating of segmentation principles can be helpful in driving promotions, formulating pricing strategies, developing marketing campaigns and in overall, achieving better relationships with customers, partners and vendors. A significant setback for retailer is when they attempt to categorize and treat customers on the same premise. Segmenting enables retailers to have a better and more realistic perspective of consumer behavior, and consumption patterns and trends, thereby developing the ability to strive for greater loyalty by effectively communicating to the defined segment. By using Table Analysis Tools for Excel, business analyst is able to create easily identifiable and understandable segments that are representative of are representative of their customers. Figure 4 provides a sample customer demographic data for customers that have made a bike purchase from a retailer in the recent past. By segmenting these customers, the bike retailer can achieve a more realistic picture of the different classifications of customers who are likely to make a repeat purchases in the future. Such information can be used to develop future campaigns for cross selling. The retailer is better able to understand the customers spending habits, and better decisions can be made regarding how marketing resources can be made. Figure 4: Sample of democratic data (2008) The chart in Figure 5 is a typical example of results that clustering can achieve. It contains a perfuming category analysis by using The Table analysis tool. The tool made it possible to automatically create six categories of customers. The tool provides an option of either automatically defining the number of categories or letting the end user decide the number of categories are appropriate. After creating categories, the end user can then determine any meaningful correlations that are encapsulated by the category. Since the focus of the retailer is on customers who have either purchased or failed to purchase a bike, it is only necessary to explore the categories that favor or disservice the purchase of the product. Figure 5: Initial results from category analysis (2008) A more detailed and segmented outlook at customers who have made a purchase is illustrated in figure 6. As suggested in the graph, category 1, 3 and 4represent customers that have made the purchase. It is then possible for the bike retailer to further explore the customers segments that have been generated in order to get a proper picture of the overall attributes of customers that belong to these categories. Of interest to the retailer would be the demographic data that influenced the creation of these segments. Such information can be useful in targeting new customers or establishing the reasons why certain stores may be underperforming. Figure 6: A detailed look at the detect categories results (2008) The significance of segmentation processes is that retailers are enabled to a better comprehension of their customer’s attributes and expectations. Segments, however, are subject to continual change, since consumer patterns and behavior change over time. This requires a recalculation of the segmentation analysis from time to time to check for correctness. A description of the details of category 3 is captured in Figure 7. Figure 7: A detailed description of category 3 (2008) 4.3 Shopping Basket Analysis Shopping basket analysis entails understanding products that comprise a customer’s order, or ‘shopping basket’. This form of analysis is often used in retail environments to understand customer tastes and preferences. Shopping basket analysis provides information that is useful in cross selling, creating product packages that are often purchased together to enhance customer satisfaction and up-sell, and develop offerings that combine high margin products with high demand products to increase sales revenues. By leveraging insights gained from shopping basket analysis, retailers can be able to optimize their assortment planning and validate their promotions. After making purchases at checkout, customers can be offered with relevant marketing information or product recommendations that can influence future purchases as well as overall customer satisfaction. The results of this analysis deliver an understanding of existing purchasing behavior in addition to effective recommendations for future. The data provided in Figure 8 represents typical data requirements that are traditionally used to undertake a market basket analysis. Analysts undertaking this procedure are required to have a set of variables that represent the transaction, the products purchased during the transaction, and probably a value that represents the transaction, such as a profit or cost depending on how the problem is defined. Figure 8: Detail data for performing a shopping basket analysis (2008) As illustrated inn figure in Figure 8, two reports are presented after the Basket analysis wizard is run. The first is the bundled shopping basket items, and the second is customer purchasing behavior. Retailers use this information to make decisions regarding pricing strategies, promotions and merchandising initiatives. The second report illustrated in Figure 9 provides users with probable recommendations. This information can be used when deciding which products should be recommended to customers, and provides an insight on how each recommendation is bound to affect the total value of the transaction. Figure 9: Suggested recommendations (shopping basket analysis) 4.4 Demand Planning The impact that accurate demand planning has on customer satisfaction and future consumption patterns cannot be overstated. In the event that a retail store experiences a stock out, this causes a decline not just in sales, but in customer satisfaction. In contrast, overstocking an outlet with a particular product sends the message that managers of the store either have no regard for their customer’s needs, or are limited in their understanding of market requirements. By integrating demand planning into operational, merchandising and store processes can be helpful in creating a more efficient system. Figure 10 contains sales data extracted from multiple global regions. The first column provides the time series dimension. It contains each sale’s year/month timestamp. Figure 10: Forecasting wizard with Microsoft data mining (2008) Analysts can apply the forecasting capabilities of SQL Server 2008 to predict sales for a specific duration as illustrated in figure 11. Figure 11: Visual display for forecasted values (2008) As illustrated in Figure 11, not only are analysts provided with a visual representation of predicted values, the values can also be written back to the original Excel worksheet. This enables analysts to easily switch between exploration and exploration, serially providing forecasts that can be incorporated back into business decisions. 5.0 Conclusion As illustrated in this paper, the influencers of specific activities such as loyalty and churn ca n easily are identified from historical customer databases by using Data Mining Add-Ins for Excel. These can be used to solve multiple organizational problems. In this paper, the retail environment was used to explore tool used in predictive analysis, but this methods are applicable to other managerial functions such as Human Resources, Supplies Management, logistics optimization, customer value and price and promotional analysis. Predictive analysis and data mining are not new concepts. For many years traditional statistical techniques have been used to model trends in customer behavior, predict stock prices, model exchange rates and assess labor requirements. But with the development of data mining software, predictive analysis is now becoming more incorporated and engaged in determining business decision and processes. The setback to this new development is that more often than not, data mining and statistical analysis is sidestepped because of the unfamiliar user environments they occur and the fact that users are required to have statistical understanding and programming expertise 5.0 Bibliography Berry, Michael J.A., and Gordon L., Data Mining Techniques For Marketing, Sales and Customer Support, Wiley, 1997 Groth, R., Data Mining: a Hands-On Approach for Business Professionals, Prentice-Hall, 1998 Hosmer and Lemeshow, Applied Logistic Regression, Wiley: 1989 SQL Server Web site: http://www.microsoft.com/sql/technologies/dm/default.mspx SQL Server TechCenter: http://technet.microsoft.com/en-us/sqlserver/default.aspx SQL Server DevCenter: http://msdn2.microsoft.com/en-us/sqlserver/default.aspx Read More