The Most Commonly Used Evaluation Methods - Case Study Example

Introduction Evaluation is a method of enquiry that documents the formation, implementation and outcome of a service or program (Jackson, Williams & Elliott, 1996). An evaluation occurs within a specific context and the program managers play a crucial role in framing and describing that context to evaluators. Evaluations if conducted properly can increase knowledge about a particular facet, reaffirm benefits, indicate lines of improvement, and enable strategic decision-making. Even if a program is successful evaluation helps to keep the program on track by identifying the changing needs more quickly. This requires understanding of objective data and subjective intuitions. Each has strengths and weakness and is helpful in management decisions. Objective date provides the most credible information from the point of external reviewers but at times certain data may be missing in which case subjective intuition is helpful. In the organizational context several methods of evaluation have been developed and this paper will address the most commonly used evaluation methods. SWOT Analysis In an organization decisions and actions have to be taken by the management in consultation with all levels within the organization. Many approaches and techniques are used to analyze and take decisions. SWOT analysis evaluates the Strengths, Weaknesses, Opportunities and Threats to an organization. SWOT analysis is an important support tool for decision making which helps an organization to build strategies or exploit strengths and overcome threats (Yuksel & Dagdeviren, 2007). The strengths and weaknesses are analyzed by an internal environment appraisal while the opportunities and threats through external environment appraisal. These strategic factors are summarized by the SWOT analysis. Comprehensive environmental analysis helps to recognize the internal and external forces with which an organization is confronted. It also allows comparing the internal and external factors. These forces may carry potential stimulants or potential limitations concerning the objectives that the firm wishes to achieve. The obtained information is systematically represented in a matrix which helps the firm to determine the strategic process. Although SWOT analysis is a good base for strategy formulation it has its own weaknesses. The magnitude of the factors is not quantified to determine the effect of each factor on the strategy adopted. This implies that SWOT analysis does not provide an analytical means to determine the relative importance of the factors. The individual factors described are general and brief. Even though internal and external factors can be compared, the result of SWOT analysis remains an incomplete qualitative examination of the internal and external factors. Hence this method is not able to comprehensively appraise the strategic decision-making process. PEST Analysis To scan the external macro environment in which a form operates, strategy consultants use PEST analysis. PEST is the acronym for political, economical, social and technological factors. PEST plays an important role in the value creation opportunities of a strategy. Macro-economical factors can differ across continents, countries and even regions and hence PEST analysis should be performed individually. A firm operates within the framework of a political and legal environment which includes the government rules, laws and regulations. It is also governed by the tax regulations and the international trade policies and restrictions. It has to abide by the safety regulations and the employment laws. The economic factors include government spending, the unemployment policy, the exchange and inflation rates, the stage of the business cycle, the consumer confidence that the firm enjoys and the overall economic growth. Income distribution, demographics, population growth rates, lifestyle changes, education, work, career attitudes, health consciousness and welfare and the living conditions fall under the purview of the social factors. Government research spending, industry focus on technological effort, new inventions and development, rate of technology transfer, energy use and costs, changes in technology, also have a bearing on the analysis which help the consultants to arrive at a strategic decision for future growth. Environmental analysis has to be continuous and must feed all aspects of planning. PEST analysis is a simple but widely used tool as it ensures that the decisions are aligned positively with the powerful forces of change in the world (Mindtools, 2007). PEST analysis also helps avoid wrong decisions PSET analysis is particularly helpful when starting a new venture as it helps to adapt to the realities of the new environment. It operates in three stages which include identifying the relevant factors, collecting information relevant to these factors and finally drawing conclusions. It is a good tool for understanding the environment in which the firm operates and becoming conscious of the threats and opportunities that face an organization. Kirkpatrick’s four-level model Decision-making is required at all levels and in all types of organization. To determine the success or to evaluate the appropriateness or usefulness of learning process, Kirkpatrick’s four-level model for evaluation can be utilized. It incorporates all major aspects of training programs and includes reactions, learning, transfer and results. According to the model, the evaluation has to start at level 1 or reactions and proceed sequentially upwards whenever appropriate, keeping the budget and time frame in mind (Porthin, 2003). The results of each level serve as the basis for evaluation at the next level. Thus each level gives a deeper picture of the training program but it also requires more rigorous and time-consuming analysis. At the reaction level questionnaire are used to determine the feelings of the learner, the opinion about the course material and the course method used apart from the appropriateness of the course itself. At level 2 or the learning stage, the knowledge attained or the change in values is determined. This assessment is usually done through a test right after the course and the questions are so designed to measure how well the learning objectives are being met. This assessment does not indicate whether the participants are able to apply the knowledge so attained in their normal working environment. At the third level, the assessment determines how the knowledge, skills and attitude changes are transferred from the learning environment to the everyday work environment. If the knowledge acquired is not utilized at the work place, the organization does not benefit by giving this training. This assessment is usually done 6 months to a year after the learning has taken place. Tests, surveys and interviews are the means through which the changes are observed. At the fourth level the results are measured in terms of increased production, improved quality, decreased costs, increased sales or higher return on investment. From an organizational perspective this is the overall reason for training. This model assesses the need for training professionals and is a straightforward system for talking about training outcomes. This model simplifies the complex process of training evaluation. However, this model has certain limitations. It is said to be incomplete as it does not take into consideration the culture of the organization, the organizational goals and values, the climate for learning transfer, the tools and equipment required for learning (Bates, 2004). Although these are considered essential for effective evaluation, this model does not consider them to be so. This model further assumes that each level will provide more information than the previous level but the data generated from this model does not provide an adequate basis for this assumption. Value chain analysis According to Porter, every firm is a collection of activities that are performed to design, produce, market, deliver and support its product. All these can be represented using a value chain. A firm’s value chain reflects its history, its strategy, its approach to implementing strategy, and the underlying economics of the activities themselves (Hales & Barker, 2003). It describes the generic activities undertaken by the firm to add value to the products and services delivered to the customer. The primary activities are concerned with the transformation of the input into output while the support activities include the activities that support primary activities like procurement, human resources and infrastructure. Through the value chain all these activities can be analyzed individually or collectively to determine the opportunities for competitive advantage. Competitive advantage can be increased if the linkages between the various elements can be exploited. The value chain is important because the organization and function of the work elements should reflect the way the firm creates and provides value for customers. There have been efforts to align it more closely with contemporary business conditions. Balance Score Card The balance scorecard (BSC) is another tool, a performance measurement framework that provides an integrated look into the financial and non-financial business performance of a firm (Lee, Chen & Chang, 2006). As the name suggests, balance is maintained between the short- and long-term objectives, between financial and non-financial measures and between internal and external performance perspectives. The four perspectives that are measured include financial, customer, internal business process and the learning and growth. The financial perspective usually relates to profitability and the measurement criteria are cash flow, ROI, return on invested capital (ROIC) and economic value added (EVA). To determine the market segmentation, customer tastes and expectations, customer retention and new customer acquisition are essential for the company to take strategic decisions. To measure internal business process value chain analysis helps. Internal value chain assesses innovation, operation and after sales service. Employee growth is an intangible asset to firms and hence it is essential to measure employee satisfaction, training and skills and continuity through turnover rate, expenses on training and lead time for introducing innovation into the market. Three principles have to be adhered to in measuring BSC – maintaining cause and effect relationships, comprising sufficient performance drivers and keeping a linkage to financial measures. BSC is the basic frame which needs to be customized depending upon the industry and the firm. The number of perspectives may be enlarged or replaced depending on the sector. The BSC concept can also be applied to measure, evaluate and guide activities in specific functional areas of a business and even at the individual project level. It has been adopted as a foundation for strategic management system and helped managers to align their business to new strategies towards growth opportunities. BSC software programs can also be used to extract data from computer based information system to obtain required performance indices. BSC too has its limitations and drawbacks. Even though the framework tackles performance at several levels, it has disadvantages and pitfalls. First of all there are no generic measures that fit all organizations in the same way. Secondly, there are a variety of quantitative indicators and BSC does not consolidate these performance values. The consolidation has to be carried out intuitively. Conclusion Thus we see that various methods of evaluation are available and it is up to the strategy consultant to use the right methodology. Each evaluation method has its own advantages and disadvantages and its application would differ across industry. For instance, the BSC is extensively used investment decisions in the IT/IS sector while SWOT and PEST analysis are mostly used to evaluate the market and to make market segmentation smoother. The SWOT analysis too determines a firm’s strengths and weaknesses which enable taking strategic decisions. The firm can then decide to exploit its strengths and gain competitive advantage. The BSC determines the balance that a firm maintains between its short- and long-term goals and objectives and the planning that it has done. It measures performance across four different perspectives each of which is important for enhancing business processes. The Kirkpatrick’s four-level model is most often used to evaluate the training programs as the ultimate result is that the forms should gain from training programs. It also assesses the need to carry out training programs and highlights the benefits that a firm can expect. While all the methods have tried to be comprehensive but at every stage innovation has been introduced to keep pace with the changes that take place in the work environment. Value chain analysis reflects the history and the strategy of the firm and helps determine the competitive advantage of the firm. it analyzes how a firm can add value to a service or product before delivery to the customer. A straight comparison of the methods is not possible as each evaluation method has to be used for a different purpose and could also vary across industry. References: Bates, R., (2004), A critical analysis of evaluation practice: the Kirkpatrick model and the principle of beneficence, Evaluation and Program Planning 27 (2004) 341–347 Hales, K., & Barker, J., (2003), Value Creation and the Virtual Enterprise, 14 July 2007 Jackson, K., Williams, K. R., & Elliott, D. S., (1996), Program Evaluation Overview, 14 July 2007 Lee, A. H. I., Chen, W., Chang, C., (2006), A fuzzy AHP and BSC approach for evaluating performance of IT department in the manufacturing industry in Taiwan, Expert Systems with Applications 34 (2006) 96–107 Mindtools (2007), PEST Analysis Understanding "Big Picture" Forces of Change, 14 July 2007 Porthin, M., (2003), Comparative Analysis of Learning Approaches in a Course on Decision Analysis, 14 July 2007 Yuksel, I., & Dagdeviren, M., (2007), Using the analytic network process (ANP) in a SWOT analysis – A case study for a textile firm, Information Sciences 177 (2007) 3364–3382 Read More