Research Methodology for User Evaluation - Dissertation Example

?Chapter 9: Research Methodology for User Evaluation This chapter will explain the steps taken to analyze and evaluate the new proposed model “Unified Theory of Acceptance and Use of Technology (UTAUT)”: The user evaluation process involved 6 stages, discussed below; 1- The first stage was to design a prototype system (UbIAMS) and to formulate the five scenarios which explain the concept of UbIAMS. 2- Survey designing and development. 3- Survey, prototype systems and five scenarios Pre-Testing (Informal pilot study) 4- Selecting some user experience benchmarking (Qualitative Dominant) 5- Solicit user participation 6- The last stage was to obtain evidence and conduct the results from the analysis of the survey, which will be elaborated in Chapter 10. The following sections give a more detailed explanation for these 6 stages that had been undertaken. 9.1 The Development and Designing a prototype system (UbIAMS) and the five scenarios The relevance of a prototype in the development of a system cannot be undermined. It bears even greater relevance when the concept of the system is innovative and has not been explored before. The premise of UbIAMS is innovative therefore the development of its prototype was achieved after extensive analysis of the features that should be included in it. The prototype system is available at www.ubiams.wordpress.com. The development of the scenarios has been achieved by analyzing the basic steps of operations that would be undertaken by any user to evaluate the extent of the functionality of the system. The scenarios have been designed after considering the level of computer skills, experience and age range of different types of users. Scenario 1: The first scenario is about creating a new account in which his login credentials, preferred authentication tools and other information shall be registered in the system. Scenario 2: This scenario involves the usage of the USB, that possesses the synced ID, to be used as the authentication tool in the system. Banking services are being used in this scenario. Scenario 3:This scenario involves the usage of the government services by demanding the presentation of the biometric features of the user. The user has a number of options to choose in terms of biometric authentication. Scenario 4: Fourth scenario involves making changes in the personal settings of the user to protect his personal infroamtion and to enhance the level of privacy from any undesired source. Changes may also be made to alter the settings for font type, color schemes, languages etc. Scenario 5: This scenario involves the usage of the system on an alternate technology and platform to analyze the level of compatibility with multiple environments. A video has been designed to provide a means for visual recollection of all the processes and functions of the system. It is common for people to understand computer operations in a better manner if visual step-by-step tutorials are available. The video shall be made available in both Arabic and English languages. 9.1.1 Profiling of exemplar UbIAMS The preceding chapter explains the models that form the basis of the development of UbIAMS. The addition of the word ‘ubiquitous’ has transformed the conventional concept of being constrained at one place to being available in different spaces; this introduction enhances the application to physical, as well as virtual spaces. UbIAMS has been designed on the basis of multiple models, including, architectural model, behavioural model, interaction diagram, collaboration diagram, sequence diagram and state machine diagram. The three criteria, security and identity, accessibility and acceptability, usability and user experience have been highlighted in the architectural model to ensure the inclusion of a diverse range of properties in the end product. The behavioural model has been developed for UbIAMS to highlight the basic operation of the system. Collaboration and interaction diagram can help the user evaluate the complexity and usefulness of the operations. State machine and sequence diagram also facilitate the inclusion of smooth functions in the system. 9.1.2 UbIAMS ( more details about the features of the system) The accumulation of a diverse range of authentication modes in a single system has been an innovative direction that is expected to become the benchmark of systems and technology in the future. The inclusion of USB, biometric features and smart cards enhances the level of usability, security and accessibility for the users. The effectiveness, efficiency and robustness of the system on a variety of platforms and technologies tend to improve the level of applicability on different types of gadgets, thus making it more acceptable and convenient for the users. The major concern of keeping information protected from undesired sources is also available in the system by giving the freedom to reveal or conceal information from any source, information flow or organization. Usually, systems are fixed with the settings that have been developed by the programmers of the system. This makes the system rigid for the users and may prove to create hurdles in the process of effective functioning of the system by the users. UbIAMS provides effective and easy change in settings that cause changes in the colors, font, language etc of the system thereby making the system flexible for all types of users. 9.2 Survey Development and Design 9.2.1 Design the Instrument (questionnaire) The design of the research instrument consisted of three pages and a cover letter, which indicated the purpose of the study and contact details for the researcher and the supervisor. The measuring instrument consisted of four parts. Part one and two covered the demographic variables (age and gender). Part three covered respondents’ computers and Internet knowledge level of experience. Part four covered the empirical measurements for the constructs in the proposed model. Sekaran and Bougie stated that the initial step in the measurement of the construct is to define the premise and scope of the construct that is desired to be measured [a]. Kholoud also attempted to explain the difference between construct and variable and stated that measurement is the only factor that can be used to distinguish between them [b]. Kholoud quoted Sekaran and Bougie [a] and explained that the presence of an actual measure changes the construct into a variable. 9.2.1.1 Created response item The questions used to evaluate the IAMS have adopted a Likert Scale. The Likert scale is a commonly used approach in questionnaires to measure participants’ opinions and attitudes regarding a certain statement. The different scales are known as Likert items, and are used instead of numerical scales (Lankes, 2002). The response option adopts a five-point scale format, with the Likert items and their weights used in the questionnaire shown in the following Table. Table: The Likert items and their weights Opinion Weight Strongly Disagree 1 2 3 4 Strongly Agree 5 9.2.1.2 Design online questionnaire The research questionnaire was formulated with the aim of measuring UX for using IAMS in physical and virtual spaces. It constitutes a set of questions, and has been made available online1 . 9.2.2 Instrument Translation into Arabic Language Back translation is a procedure commonly used in cross-cultural survey to test the accuracy of the translation (Brislin, 1970 & 1980)2. The research instrument with its valid set of items will be translated from English to Arabic by an accredited bilingual. 9.2.3 Design a video tutorial for using the UbIAMS A video tutorial contains the steps to use UbIAMS 9.3 Survey, prototype systems and five scenarios Pre-Testing (Informal pilot study) A sample of 8 Lay Experts were selected (Rubio, 2003)3, all of whom are postgraduates or researchers at the Electronics and Computer Science School at the University of Southampton, and have had experiences in one of the research dimensions. There were 5 females and 3 males, ranging in age between 20 and 50 years. Importantly, the sample members were recruited based on their interest in the research perspectives. This step was implemented in an attempt to garner comments and recommendations from lay experts concerning the questionnaire design, and to further ensure the users doing the evaluation were able to understand the five scenarios, the prototype systems and the instrument. After meeting each lay expert individually, comments were provided, with some questions recommended for deletion, and the phrasing of some statements highlighted as being unclear. After taking their feedback and making the necessary changes, a total of 60 questions were created for the instrument. The final version of the questionnaire is attached in the file named (Questionnaire.docx) The Arabic version will be checked by PhD students for terminology checking. The final copy will be compared to the original instrument to check validity of the translation process. After translation will be completed, a field tests was carried out to ensure comprehension of all questions by subjects. 9.4 Selecting some User Experience Benchmarking (Qualitative Dominant) Validity looks at how well the items of an instrument represent a concept or domain of content (namely "Content validity"). Validation becomes an important step especially when a new measure is being developed where there is no existing measure that operationalizes the concept as the researcher intended. For example, there are instruments measuring security, accessibility and usability in general but not in identity access management systems; moreover, a model that defines all of these together in identity access management systems or an instrument that measures user experiences in identity access management systems is new and need to be validated. Using a panel of users during a validation will provide useful feedback about the quality of a newly developed measure. Without conducting a validation study, a researcher is using an untested measure to conduct their study. Data from an untested measure may indicate that the instrument needs revisions and the process would need to be redone with another pilot study for the revised instrument. Hence, more resources and effort will be spent in evaluating and recreating the instrument. Content validity is assessed using qualitative techniques. In the development of multi-item rating scales, the content validity of the questionnaire items will be examined by using users from different ages, genders and experiences. Quantitative research in the form of online and offline questionnaire survey and Qualitative research will be undertaken in order to answer the selected hypotheses. This study will use equal status ‘Quantitative and Qualitative Dominants’. From the Quantitative Dominant, questionnaires shall be distributed amongst users who shall provide their opinions concerning the different features of the prototype system (UbIAMS), as well as their own level of inclination to utilize technology in their lives. The pre-generated questionnaires, User experience methods and HCI approaches shall cover all the aspects related to the diverse range of users that are aimed to be addressed with this technology. From Qualitative Dominant, this study will use some User Experience Benchmarking such as: Task success, Time on task, and Error rating. 9.5 Solicit user participation 9.5.1 Samples Descriptions a. Identify users from different ages, genders and experiences as participants. Selecting the sample and calculating the sample size for providing the required confidence in the answers. This will be done using GPower Software4. b. After conduction an evaluation using the lay experts, an invitation email requesting users’ participation will be send two week before the actual start of the study; this allowed enough time for the individuals to respond to the request. The invitation email included a cover letter detailing: the purpose of study, a description of the measure and its scoring, and an explanation of the response form, a link to the research website, notably containing a brief about the study and a video tutorial, and ethics information, and Doodle link that contains list of seminars date and place. If they agreed to participate, they should select the best time and place for attending the seminar. 9.5.2 Participations The study will be limited to Saudi citizens who use the physical and virtual public services provided by the Saudi Identity Programme at the National Information Center under Ministry of Interior of Saudi Arabia. The acceptance letter for doing and testing the current research at the National Information Center in Saudi Arabia are shown in Appendix 9.B. 9.6 Obtain evidence and conduct the results from the analysis of the survey 9.6.1 Ethical Considerations Participates don't need to provide any personal information or any authentication tools because they will do five scenarios that contain all the information that they need it. For sure, all of their answers will be kept completely anonymous and confidential. Moreover, the data will be destroyed and deleted after analyzing the answers and evaluating the model. 9.6.2 Evaluation process Two week prior to the commencement of the study, an email will be send to users, inviting them to participate in the study. If they agreed to participate, they should select the time and place to attend the seminar. The sample is informed that their participation is voluntary, and that all data are completely anonymous and confidential. The study is on-going for two months. The procedure will be as follows: 1. Participant receives an invitation email containing the steps of the user evaluations and electronic consent form uploaded on the research website . The email advised of the study deadline. The individual will be able to browse the research website and watch the video tutorial < http://ubiams.wordpress.com/help-and-video-tutorials/ >. 2. If participation is agreed, they should select the time and place for attending the seminar using Doodle link < http://www.doodle.com/kw7x5mbxtckenyka > , and they have to send the consent form after they fill it. 3. In the date of the seminar, the researcher will describe to the participant the research study, and instructions for doing the five scenarios, and filling the instrument online. (around 5 min) 4. If participation is agreed upon, the participant should do the five scenarios < http://ubiams.files.wordpress.com/2012/11/scenarios.doc > using the prototype system. Then, they have to click the instrument link , and then the ‘Next’ button will be clicked. If he/she declines, the ‘Back’ button will be clicked, or the browser can be closed to exit. (around 25 min) 5. The participant will then answer by clicking on the radio buttons. 6. After completing the questions, they are instructed to click the ‘Exit’ button or to close the browser to exit. Materials The Questionnaire is attached in the file named (Questionnaire.docx) The link of the Video Tutorial and Video Transcript are attached in the file named (Video.docx) The link of the prototype system which is available online in The five scenarios are attached in the file named (Scenarios.docx). 9.7 Chapter Summary The formulation of the considerations for the development of the prototype has been discussed in this chapter, alongside the scenarios that shall prove to facilitate the functionality of the system for the users. The scenarios also provide a full insight into the system and the scope of its functions before even operating it. This chapter also described the process and design of the survey which is an important step in the acquisition of feedback from the users. Different age groups and types of users (varying levels of experiences and diverse degrees of disabilities) have been chosen to ensure that the end product proves to be flexible for all ages and user groups. Developers often develop systems without considering the probable elementary computer skills of users therefore this factors has been given careful consideration in the designing of this system. This chapter also highlights the ethical considerations that have been undertaken in the processes of acquiriung feedback from the users. Their willingness to participate in the study and eagerness to protect their information have been kept as top priorities. The users that have been chosen for the research study have been limited to the land of Saudi Arab to reflect a true image of the level of acceptance that can be expected from the region after its eventual launch. The next chapter shall include details regarding “Statistical Tools and Data Analysis Approaches”. References [a] U. Sekaran, R. Bougie, “Research Methods for Business- Fifth Edition”, John Wiley and Sons, 2009. [b] Kholoud paper Read More