Evaluation methods for the proposed IAMS framework, model, and system - Thesis Example

Table of Contents (to show the outline of the methodology chapter) Table of Contents (to show the outline of the methodology chapter) 1 INTRODUCTION 1 LITERATURE REVIEW 1 MethodologY 2 What the Evaluation Methods Should Achieve 2 Background to Evaluation Methods 3 Methods of Evaluating Expert and User Opinions 3 Empowerment Evaluation Method 3 Deliberative Democratic Evaluation Method 4 Realistic Evaluation Method 4 Utilization-Focused Evaluation Method 4 Methods of Evaluating System Performance 5 Expert Judgment/Peer Review Evaluation Method 5 Use-Case Based Evaluation Methodology 5 Comparison of the Different Evaluation Methods 6 Comparison of Expert and User Evaluation Methods Using the Three-Group Typology 6 Comparison of System Performance Evaluation Methods 8 A Tabular Summary of the Evaluation Methods 8 Evaluation Method Selection in the context of the Scope of the Research 9 Evaluation Method Selection Criteria 9 Selection of the Most Appropriate Evaluation Methods 11 Reasons for selection of Realistic Evaluation Method for Expert Opinion 11 Reason for Selection of Empowerment Evaluation for User Experience 13 Reasons for selecting Expert Judgment for Analysis of IAMS Performance 14 Statistical Analysis Experts’ Opinions using SPSS 15 ANALYSIS 16 DISCUSSION 16 CONCLUSION AND RECOMMENDATIONS 16 References 17 INTRODUCTION LITERATURE REVIEW MethodologY This chapter explores evaluation methods for the proposed IAMS framework, model, and system. The underlying aim of the methodology is to identify the most appropriate method for evaluating experts’ opinion collected through the questionnaires to achieve the aforementioned research objectives. In an effort to attain this agenda, different evaluation methodologies are undergo careful analysis with the aim of identifying the most appropriate evaluation method for each of the three components under examination: expert opinions, usability evaluation, and system performance. Each of the three components seeks to improve the proposed IAMS system in a specific aspect. The expert opinion aims to develop a strong framework for the IAMS system, the user evaluation seeks to assess the appropriateness of the proposed IAMS model, while the performance evaluation will reveal the performance capacity of the new system. What the Evaluation Methods Should Achieve The appropriate evaluation method should reveal the most appropriate elements that would provide user opinions for an acceptable integration model for the physical and virtual identity management systems. In addition, the evaluation method should elucidate the most comprehensive and efficient IAMS framework for the integration of physical and virtual IAMS. Finally, the evaluation methods should have the capacity to test the components that constitute the conceptual model, so that the final model merges seamlessly with user expectations and usability preferences. Background to Evaluation Methods Good theoretical frameworks offer practicality in a wide range of situations, and therefore their importance in evaluation of IAMS frameworks cannot be overemphasized (Alkin, 2004). According to Alkin (2004), evaluations stem from social responsibility, monetary considerations, and social inquiry. In explaining the evaluation methodology, Alkin (2004) uses a comprehensive categorization based on use, methods, and value of the evaluation components. In this paper, three distinct evaluations are necessary to achieve the objectives of this research: expert evaluation, user evaluation, and evaluation through comparison with existing IAMS frameworks. Methods of Evaluating Expert and User Opinions Empowerment Evaluation Method An empowerment evaluation method would focus on helping a particular group of stakeholders, either clients seeking service through the IAMS or providers of the IAMS services. In addition, empowerment method takes into consideration feedback from different stakeholders (Fetterman and Wandersman, 2007). Expert evaluation with empowerment method therefore gives evaluators more responsibility in a supervisory role than in an expert role. Methods that can use of empowerment evaluation include analysis of expert questionnaires, such as those collected for the purpose of this research (Fetterman, 2007). The overriding premise of empowerment evaluation is to help people contribute to what helps them carry out their roles (Sherriff & Porter, 2010). Deliberative Democratic Evaluation Method Deliberative democratic evaluation seeks to arrive at unbiased evaluative conclusions through engagement of the main stakeholders, and sometimes employs the use of other evaluation methodologies (House, 2003). Democratic evaluation seeks to maximize the social value of the IAMS framework under evaluation, with regard to the social implications of the suggested IAMS framework. Deliberative democratic evaluation relies on evaluation methods such as expert judgment, which is a low cost, fast application, and wide acceptability; and injects experience and knowledge into the evaluation process (Ruegg & Jordan, 2007). Realistic Evaluation Method Realistic evaluation focuses on the preference and usability context with regard to the manner of use of the system under evaluation (Westhorp, 2011b). Realistic method of evaluation not only considers what work, but for whom and in what way and in which context (Westhorp, 2011a). Through realistic evaluation, it is possible to establish on whom the responsibility of implementing the new IAMS framework falls and consider these factors in the analysis. Realistic evaluation method can help in designing an IAMS framework with a bent on the sources and verifiers of identity information for both highly debated issues and less contentious issues. Utilization-Focused Evaluation Method Utilization-Focused evaluation perpetuates the belief that evaluations are because of usefulness of the system to the end-user (Patton, 1997). Therefore, an IAMS model based on this evaluation methodology would disregard the needs of the intermediaries and collaborative sources of identity information and give greater regard to user needs and opinions. If the user opinions, rather than expert opinion, on the framework were an influencing factor to the change in the existing IAMS framework, this method would be supremely important. Methods of Evaluating System Performance Expert Judgment/Peer Review Evaluation Method The use of expert judgment evaluation is critical in improving and establishing the chances of success of new informational systems (Ruegg & Jordan, 2007). The method is particularly important due to its ability to assess system performance in a low-cost, easy-to-apply means in numerous areas of the system development life cycle. Additionally, the use of expert judgment is common for analysis of government, industry, and academic information systems. Consequently, given that the application of the expert judgment is informal, for instance, in this research, where questionnaires are sent to experts for evaluation, its administration procedures are considerably straightforward and convenient. The questionnaires used to collect expert information also give allowance for the experts to offer their opinions. Therefore, the questionnaires will help in tapping into the vast experience of the experts and benefit from their informed comments. All this input will offer considerable benefit in creating a highly effective integration model for virtual and physical IAMS with the developed evaluation framework. Use-Case Based Evaluation Methodology Use-Case based analysis has numerous fields of application, and boasts of a wide usage in software and system engineering (NSS Labs, 2008). Case-based evaluation and testing is used in testing performance in technological products, and could be appropriate for evaluating the performance of the proposed IAMS system. The method, however, places a huge focus on assessing the abilities of a system for a target market segment, with the aim to match the needs of the buyers of the system. The evaluation method implies that its use may be a little inappropriate for the system proposed in this research, which targets government and industry players in the field of IAMS. Comparison of the Different Evaluation Methods Comparison of Expert and User Evaluation Methods Using the Three-Group Typology The four methods of evaluation in the above discussion are analysed using a three-group typology. The three-group typology focuses on three groups of people: users, producers, and intermediaries in the IAMS framework. According to Contandriopoulos and Brousselle (2010), producers supply the confirmatory information, which help in the analysis of the IAMS framework. The intermediaries act as a link between the source of this information and users, who are the clients and customers who require the IAMS system to get their needs attended. The typology is born of the realization that different stakeholders have a stake in the IAMS framework and model The three-group typology also takes into consideration the level of consensus among stakeholders with regard to the proposed changes to the IAMS framework and model. High level of disagreement represents high polarization while low level of polarization represents low polarization. The diagram below (Brousselle & Contandriopoulos, 2010), displays the outlay of different evaluation methods with particular levels of polarisations. Immediately below the diagram is a summarised table version of the graphical representation (Contandriopoulos & Brousselle, 2012). Evaluation Methods (Brousselle & Contandriopoulos, 2010) Below is a comparable tabular summary of the above diagram Needs Prioritized by the New System Evaluation Method Level of Polarisation Low High User-focused Empowerment Evaluation Yes Partly Utilization-Focused Evaluation Yes No Producer/Intermediary focused Realistic Evaluation Yes Yes Deliberative Democratic Evaluation No Yes The level of polarisation, which can be either low or high, indicates the applicability of each of the four evaluation methods under each of these polarisation levels. The green areas show in what areas the user-focused evaluation method will be most appropriate for the IAMS model. Therefore, both empowerment and user-focused evaluation would be highly relevant for evaluating user-focused IAMS component with little level of controversy. The blue areas show areas where the proposed IAMS framework will be most applicable with regard to the intermediaries and sources of the IAMS information. Evidently, it is apparent that realistic evaluation would serve evaluative procedures for both highly polarised and lowly polarised components sufficiently. On the other hand, deliberative democratic evaluation would be restrictive, and usable in evaluation of highly polarised IAMS components. The grey areas indicate points where the evaluation method has little or no significant application. Comparison of System Performance Evaluation Methods Two methods of evaluating system performance have been considered in the prior discussion on evaluation methods. The methods were selected with regard to their applicability in assessing system performance. Some evaluation methods concentrate on the interests of the user and others on the critical resources necessary to run the system (Chandler & DeLutis, 2011). With regard to the research design for this study, which relies solely on expert opinions, expert judgment evaluation offers the best method of evaluating the performance of the proposed IAMS system. The use-case based method is more applicable for assessing informational systems for commercial purposes, as it focuses on creating a system that meets customer demands. The proposes IAMS system, on the other hand, focuses on meeting stakeholder requirements specifically the government standards, industry guidelines and administrator preferences with regard to IAMS systems. A Tabular Summary of the Evaluation Methods Evaluation Method Stakeholders for Method use Methods for this evaluation User Experience Methods Expert Opinion Evaluation Methods Realistic Evaluation Method Experts Email Survey, Questionnaires (Pawson & Tilley, 2004) Surveys, (Rohrer, 2008) Deliberative Democratic Evaluation Experts Email Survey,Questionnaires (House, 2003) Surveys, (Rohrer, 2008) User Experience Evaluation Empowerment Evaluation End-User Usability Testing, Customer Feedback via Email, (Fetterman & Wandersman, 2007) Usability studies, usability benchmarking(Rohrer, 2008) Utilization-Focused Evaluation End-User Usability Testing, feedback via Email (Patton, 1997) Usability studies (Rohrer, 2008) Evaluating System Performance Expert Judgment Experts Questionnaires (Ruegg, 2007) Surveys Use-Case Based Evaluation Experts Surveys and Questionnaires (NSS Labs, 2008) Usability benchmarking (Rohrer, 2008) Evaluation Method Selection in the context of the Scope of the Research The framework upon which the proposed IAMS system is modelled strives to achieve validity on many levels through a rigorous evaluation method. The evaluations intended for the new system will involve expert opinions on the proposed IAMS framework, user evaluation of the IAMS model, and comparative evaluation of the final IAMS system with existing IAMS systems. The evaluation method or methods selected therefore will be largely for helping the research achieve develop a sound framework based on comprehensive evaluation of the opinions of all the stakeholders to whom the new IAMS system relates. Consequently, the methods selected will principally focus on evaluating users and expert opinions. Evaluation Method Selection Criteria Evaluation theory acts as a guide to evaluation practice, and underscores the purpose and appropriateness of conducting an evaluation (Mark, 2005). Through the above discussed evaluation theories, it is possible to establish stratagems through which it is possible to design a successful IAMS system which wins acceptability, meets security requirements, and offers easy usability to the people in whose hands its use falls. Given that our data is qualitative in nature, and specifically relies on expert opinions as to the appropriateness of a list of numerous individual components, the most appropriate evaluation for the IAMS framework would focus on the producers and intermediaries according to the three-group typology. The reason for this is that many responsibilities for adjusting to the new IAMS fall mostly on the intermediaries and suppliers of identity information. However, the use of realistic evaluation precludes the user opinions about the new IAMS framework, and hence an additional method is necessary for evaluation of user opinions about the new IAMS. This objective is consistent with the intended scope of this study, which intends to validate the applicability of the new system in the real world through feedback from the users of the proposed IAMS model before proceeding to evaluate it with regard to other systems. For effective evaluation of user-oriented aspects of the IAMS system, the choice lies between empowerment evaluation and user-focused evaluation. Of the two evaluation methods, empowerment evaluation method offers a better choice as it can be usable for both highly controversial components in the IAMS evaluation framework and the less controversial aspects. User-focused evaluation, on the hand disregards the level of controversy aspects of the proposed IAMS framework may have, and may fail largely in validating the proposed system with such as limited theoretical outlook. Selection of the Most Appropriate Evaluation Methods Given these characteristics and considerations, the methods that stand out are democratic and realistic evaluation method, because they belong to the producer and intermediary based category. A further consideration on these evaluation methods indicates that realistic evaluation offers a wider scope given in comparison to the democratic evaluation, as it covers both high and low polarization situations, while democratic evaluation method offers little in cases of low polarization. Reasons for selection of Realistic Evaluation Method for Expert Opinion Realist evaluation is a robust evaluation technique founded on a number of principles upon which the achievement of the objectives of this research is possible. For instance, in designing the realistic evaluation method, Pawson and Tilley (2004) had a clear agenda of helping implement systems: 1. Which work for different people in whose scope the new systems reaches. The use of realistic evaluation methods will enable this research achieve a proven practical standard with regard to political, social and technical aspects of the new IAMS framework. 2. Realistic evaluation appreciates the highly important role of policy makers in helping implement new information systems (Pawson & Tilley, 2004). Such a viewpoint will be of critical importance in evaluating the framework, model, and system for the integration of the existing physical and virtual identity IAMS systems. This research makes use of expert evaluation, who bear an intimate understanding both physical and virtual IAMS systems currently in use. The strict selection criteria for the participants in the evaluation help in avoiding the very grave possibility of developing a system, which does not live up to its mandate. It is imperative that the evaluators participating in the evaluation have some intimate relationship with the use of IAMS systems, and will be instrumental to the success of the developing the proposed IAMS model for integrating the current physical and virtual IAMS systems. 3. Another overriding principle of realistic evaluation is the recognition of the importance of reconciliation of the evaluation context and proper mechanism to achieve the desired outcome (CMO) (Gill & Turbin, 2006). For the purpose of this research, usability tests and expert opinions define the context within which the new IAMS framework and model will serve, and the use of various research techniques such as questionnaires define the mechanisms with which the outcome of the research are attainable to create an integrated physical and virtual IAMS system. 4. Realistic evaluation also seeks to tend to the specific needs of the stakeholders for whom the new system will serve (Pawson & Tilley, 2004). Through realistic evaluation, it will be possible to unravel the elements that affect use of IAMS systems and help in incorporating the same into the framework design to come up with a functional and acceptable IAMS system for all stakeholders involved. 5. Finally, realistic evaluation is particular about the people who participate in the research. The use of realistic evaluation does not require the researcher to pick any group of evaluators to make comparisons. This research remains consistent with this goal by involving principally the persons for whom the final integrated system is intended. Reason for Selection of Empowerment Evaluation for User Experience Empowerment Evaluation (EE) is a widely accepted evaluation concept since its analysis by evaluation theorists Fetterman and Wandersman (Wandersman et al., 2003). There are a number of reasons for selecting empowerment evaluation for analysis of usability of the proposed framework for the integration of physical and virtual IAMS: 1. According to Fetterman (2007), empowerment evaluation is the use of evaluation concepts, techniques, and findings for improvement and self-determination. Therefore, the principles, which define this evaluation methodology, will be of critical importance in the analysis of the qualitative data and using the results in deciding whether or not the research objectives of obtaining optimum user experience. 2. Empowerment enhances the chances for success in a project (Fetterman, 2007). The evaluation framework for the proposed model of integrating the physical and virtual IAMS relies heavily on the input of the users of the system. Therefore, through intensive involvement of both users and experts, the evaluation framework will help in ensuring this research attains most of its objectives. 3. Through the involvement of users in the design of IAMS framework, the transition process for the new system will be fundamentally smoother and more welcoming to the users, as they will have the feeling that they were instrumental in its design. Empowerment evaluation of the IAMS model will engage every aspect critical to the running and implementation of the new IAMS model in a way that shows accountability and regard for stakeholder opinions in implementing a new system. 4. Many technological innovations suffer crippling failures due to lack of involvement of user feedback during the development and evaluation stage. Generally, empowerment evaluation will help in incorporation of quality, responsibility, adherence to policy guidelines, usability, and orient the design towards the most appropriate user preferences (Wandersman et al., 2003). 5. The all-encompassing principles that empowerment evaluation articulates will help immensely in collaborative planning by identification of weaknesses and improvement of the IAMS system. Through a comprehensive and widely used empowerment evaluation method, stakeholders, including the users of the proposed IAMS model will have the ability to assess, plan, implement, and self-evaluate the framework (Miller & Lennie, 2005). Reasons for selecting Expert Judgment for Analysis of IAMS Performance The appropriateness of expert judgment in evaluation of system performance for the proposed IAMS framework cannot be overemphasized. The following reasons back the use of this evaluation method for evaluating system performance: 1. The use of expert judgment is born of the realization that despite considerable amounts of feedback from other sources, the experts ultimately offer the most influential advice on performance of the system. 2. The second reason for selection of expert judgment method of evaluation for evaluating the performance of the new IAMS system is the realization that experts also possess subject-matter expertise to address the key objectives of the evaluation. For instance, in this research, some experts offer inter-disciplinary expertise on the subject under discussion for instance in the acceptability and usability segments of the framework evaluation questionnaires. 3. Finally, through the reliance on expert information for analysis of the proposed IAMS model, there will be a higher chance of success with the system in its real life application. The rigorous analysis will make the proposed IAMS system exhibit high efficiency and acceptability to meet the desired user preferences. Statistical Analysis Experts’ Opinions using SPSS Research data, according to Kitchenham (1996), can either be quantitative, qualitative or hybrid in nature. Hybrid data combines elements of both qualitative and quantitative research. Expert opinions are mainly qualitative in nature, rather than quantitative. To analyse expert opinions, statistical software, such as SPSS is necessary. SPSS is a robust statistical analysis tool, and offers immense ability to synthesise qualitative data. The statistical software allows for multiple manipulations on data, and the determination of various elements necessary for identification of significance of different components in the proposed IAMS framework. The data occurs in figures, ranging from 1 to 4, depending on the individual expert’s opinion about the component with regard to the proposed IAMS framework. SPSS can calculate average on the opinions, and determine the overall average for each of the three elements considered and determine how it fares against the existing system. SPSS statistical analysis aspects will help in filtering and selection techniques. In addition, it will be possible to calculate the percentage of experts who think a proposed IAMS evaluation framework is appropriate or not. Through the software-based analysis of data, the IAMS framework will lend itself to discussion and identification of areas in need of improvement and additions for fine-tuning the IAMS system. For instance, average for one of the three aspects of the proposed IAMS framework, security, acceptability and usability, will enable us to tell whether it comprehensively improves, maintains or performs worse than the existing system, according to the experts’ opinion. ANALYSIS DISCUSSION CONCLUSION AND RECOMMENDATIONS References Alkin, M, 2004, Evaluation Roots: tracing theorists’ views and influences, Sage Publications, Inc. Brousselle, A, and Contandriopoulos, D, 2010, Evaluation theories and evaluation use, Canadian Institute of Health Research, Pp. 11-22, viewed 15, 2012, Chandler, J and DeLutis, T, 2011, A methodology for performance evaluation of information system under multiple criteria, Educational Resources Information Center, p.1, viewed 20 April, 2012, Contandriopoulos, D and, Brousselle, A, 2010, Evaluation theories and evaluation use, The Tavistock Institute, P 70, viewed 15, 2012, Fetterman, D, 2007, Empowerment Evaluation: Principles and Practice, Stanford University, P. 15, viewed 15, 2012, Fetterman, D and Wandersman, A, 2007, Empowerment Evaluation: Yesterday, Today, and Tomorrow, American Journal of Evaluation, 28(2): Pp.179-198 viewed 15, 2012, Fetterman, et al., 2003, Empowerment Evaluation: Principles and Action, University of California, Santa Clara, Pp. 139-142, Viewed on April 16, 2012 from Gill, M and Turbin, V, 2006, Evaluating “Realistic Evaluation”: Evidence from a Study of CCTV, Crime Prevention Studies Volume 10, Pp 179-199, Viewed on April 16, 2012 from House, E. R. (2003). Evaluation theory. In T. Kellaghan & D. L. Stufflebeam (Eds.), International handbook of educational evaluation. Boston: Kluwer Academic, Pp. 9–14 Kitchenham, B, 1996, DESMET: a method for evaluating software engineering methods and tools, University of Keele, Staffordshire, Pp. 4-6, Viewed on April 17, 2012 from Mark, M 2005, Evaluation Theory or What are Evaluation Methods for? Harvard Graduate School, P.1, Viewed on April 17, 2012 from Miller, W, and Lennie, J, 2005, Empowerment Evaluation, Evaluation Journal of Australasia, 5(2), 18-26, Viewed on April 16, 2012 from NSS Labs, 2008, Evaluating Products based on appropriate usage, NSS Labs, p.3, viewed on April 20, 2012 < http://www.nsslabs.com/assets/white_papers/NSSLabs_WP_AppropriateUsage.pdf> Patton, M, 1997, Utilization-Focused Evaluation: The new century text< thousand oaks, Sage books, Pawson, R, and Tilley, N. 2004. Realistic Evaluation. Sage Books. Pp. 83-114, Viewed on April 16, 2012 from < http://books.google.co.ke/books?id=GXagvZwC2bcC&pg=PR3&source=gbs_selected_pages&cad=3#v=onepage&q&f=false> Rohrer, C, 2008, When to Use Which User Experience Research Methods, Useit.com, viewed on April 19, 2012 Ruegg, R, and Jordan, G, 2007, An overview of evaluation methods for R and D programs, U.S. Department of Energy, P. 18, Viewed on April 17, 2012 from Sherriff, B and Porter, S, 2010, An introduction to empowerment evaluation: teaching materials, P 1, Viewed on April 17, 2012 from Westhorp, G, 2011(a), Realist Evaluation: An overview, Centre for Development Innovation, P.1, Viewed on April 17, 2012 from Westhorp, G. 2011(b). Realist Evaluation. Wageningen. P.1, Viewed on April 08, 2012 from Read More