Usability Engineering - Assignment Example

Common Industry Format for Usability Test Report Comments and questions about this format: iusr@nist.gov Assignment Part 3 Usability Engineering Website (www…)

//Edit the Table of Contents
Executive Summary 1
Introduction 2
Description/Overview of site 2
Scope 2
Test Objectives 2
Method 3
Participants 3
Context of Site/Application Use in Test 3
Tasks 3
Test Facility 3
Participants Computing Environment 3
4
Experimental Design 4
Procedure 4
Participant Task Instructions 4
Usability Metrics 5
Results 5
Data Analysis 5
Data Scoring 5
Data Reduction 6
Presentation of Results 6
a) One or more measures of user satisfaction. Questionnaires to measure satisfaction and associated attitudes are commonly built using Likert and semantic differential scales. A variety of instruments are available for measuring user satisfaction (such as the SUS). 6
Example Effectiveness Results 6
Recommendations 7
References 7
Appendix A – Data Collection Forms 8
Screening Questionnaire 8
Orientation Script 8
Pre Test Questionnaire 8
Video Consent Release Form 8
Task Scenarios 8
Post Test Questionnaire 8
Data Logger 8
Facilitators Checklist 8
Appendix B – Screen Shots 8
Executive Summary

The website under scrutiny is called (DIC) and can be accessed through the URL address http://www.dreamincode.net. This website hosts an online leading community for programmers and web developers. currently has 634,778 members who are registered in the site. Each day, about 400 members register in the site (Dream in code: Online). Registered members normally enjoy full access to thousands of tutorials in programming, code snippets, and definite forums topics where members can engage in constructive discussions in regard to the topics of concern to the members. In the last eight years, DIC has become very reputable in providing expertise for many students and professional in a very friendly and timely manner. Members of this site have profiles and depending on their activities online, they are categorised as badges, contributors, authors, experts, mentors, alumni, administrators, moderators and webmasters. Badges occur below the usernames f the members. Contributors are members who submit tutorials and code snippets on the site (Dream in code: Online). Authors on the other and are members who have taken their time to write a number of tutorials or some unique code snippets. Experts are members who answer questions accurately and are seen as being experts in a particular forum. Members who are categorised as mentors are given privilege to moderate forums. The alumni are members who were previously members of the DCI staff. Moderators help the community to understand the rules of engagement and dedicate a number of hours daily in moderating the sight (HCI in the software process: Online). Administrators are awarded a green badge and are very active in the site. They help in the daily operations of the site and suggest improvements. The webmaster oversees the daily operations of the site.
Introduction
This project involves a number of issues that are related to usability engineering. It ranges from the development of the appropriate method and a detailed description of the same. This project is deeply motivated by the desire to analyse the usability of DCI and determine ways in which the target community group can be satisfied more effectively (HCI in the software process: Online). Therefore, the research work will involve two main but separate parts which are interrelated but will be very crucial in the project. First will be introduction to the site under scrutiny and its importance in the process of engineering. The next sections will be concerned with the results from the research work and the recommendations that can be made to improve community.
Description/Overview of site
DIC hosts an online leading community for programmers and web developers. It currently has 634,778 registered members. Every day, about 400 new members register in the site. Registered members can enjoy access to thousands of tutorials in programming, code snippets, and definite forums topics where members can engage in constructive discussions in regard to the topics of concern to the members. In the last eight years, DIC has become very reputable in providing expertise for many students and professional in a very friendly and timely manner.
Scope
The project will explore a number of areas in regard to the site ranging from the membership of the community, activities of the members, the interests of the community members, the kind of activities carried out in the site, the number of members in the site and the bulk of the work done by the members.
Test Objectives
To determine the computing environment in the DCI environment.
To explore the activities, membership and participation of the members of the computing community.
To determine the performance of the computing environment in terms of members satisfaction.
Method
Participants
a) The total number of participants tested.
b) Segmentation of user groups tested, if more than one.
c) Key characteristics and capabilities of user group.
d) How participants were selected; whether they had the essential characteristics.
e) Differences between the participant sample and the user population. EXAMPLE: Actual users might attend a training course whereas test subjects were untrained.
f) Table of participants by characteristics, including demographics, professional experience, computing experience and special needs. The characteristics shall be complete enough so that an essentially similar group of participants can be recruited. Characteristics should be chosen to be relevant to the product’s usability; they should allow a customer to determine how similar the participants were to the customers’ user population.
Context of Site/Application Use in Test
Tasks
a) The task scenarios for testing.
b) Why these tasks were selected. EXAMPLES: The most frequent tasks, the most troublesome tasks.
c) Any task data given to the participants.
d) Completion or performance criteria established for each task.
Test Facility
a) The setting and type of space in which the evaluation was conducted. EXAMPLES: Usability lab, cubicle office, meeting room, home office, home family room, manufacturing floor
b) Any relevant features or circumstances that could affect the results. EXAMPLES: Video and audio recording equipment, one-way mirrors, or automatic data collection equipment
c) Computer configuration, including model, OS version, required libraries or settings.
d) Browser name and version; relevant plug-in names and versions.
e) If screen-based, screen size, resolution.
f) Any hardware or software used to record data.
Participants Computing Environment
The computing configuration that was used for the test -:
Computer model and specification:
Operation System:
Internet connection:
Mouse:
Browser:
Plug-ins:
Other software:
Display Devices
The monitor configuration used for the test is as follows:
Screen size:
Resolution:
Frequency:
Colour:

Experimental Design

Include - Number of participants, number of tasks they performed, environment for each of them
Discuss –
the control variables in this test: browser, screen size, resolution, same internet connection for each participant, etc.
The measures recorded during the test per user:
Goal achievement, Number assists per task: ,Number of errors,Time-On-Task:
Satisfaction, ease of task achievement, usefulness of search functions, ease of finding way around the site, amount of information on the web pages,adequacy of the amount of instructions on the website and overall satisfaction with the website using the Likert scale.
Procedure
- Discuss the entire procedure from welcoming to thanking the users after the test
-Time limits on tasks.
- Verification that the participants knew and understood their rights as human subjects.
- Number and roles of people who interacted with the participants during the test session.
- Whether participants were paid or otherwise compensated.
-Measures that were recorded during the test per user are: discuss how you defined the measures
Example - Number assists per task: count = zero at the beginning of a task, which is incremented each time the facilitator helps the user.
-When and how the facilitator can assist and give hints must be discussed. You may use a table
Task numbers
Time limit (minutes)
Assists allowed?
Type of assistance
Participant Task Instructions
General instructions given to the participants (here or in an Appendix).
Instructions on how participants interact with any other person present, include how to ask for assistance and interact with other participants, if applicable.
Task instruction summary.
Usability Metrics

Effectiveness can be measured using -
Goal achievement:
Completion rate:
Assisted completion rate:
Error rating:
And any other
Efficiency:
Time on task:
Success in benchmark time rate:
Satisfaction:
Satisfaction measured using the Likert scale
Sample questions -
1. How do you rate your overall satisfaction of the site?
2. How useful do you think the website is?
3. How easy do you think the site was to use to achieve the tasks you were given?
…………………………….
……………………………
Results
Data Analysis
Data Scoring
Use the various metrics to discuss
Example - Goal achievement:
Task number
Completion/Performance criteria
1
100 % if the task is …
50% complete if …
0% complete if…
Satisfaction:
Metric
Measure
Scale
Measure
Overall Satisfaction
Not Satisfied
3
2
1
0
1
2
3
Very Satisfied
Data Reduction
Group tasks into types to discuss
Presentation of Results
Discuss Success rates
a) The number of participants who completely and correctly achieve each task goal.
b) Errors are instances where test participants did not complete the task successfully, or had to attempt portions of the task more than once.
c) The unassisted completion rate (i.e. the rate achieved without intervention ) as well as the assisted rate (i.e. the rate achieved with intervention) where these two metrics differ.
Satisfaction ratings
a) One or more measures of user satisfaction. Questionnaires to measure satisfaction and associated attitudes are commonly built using Likert and semantic differential scales. A variety of instruments are available for measuring user satisfaction (such as the SUS).
Example Effectiveness Results

Task type
Task 1
Task 2
Task 3
Participant 1
100
100
100
Participant 2
100
100
100
Participant 3
100
20
100
Participant 4
100
100
100
Participant 5
100
100
100
Goal Achievement (%)
Mean
100
84
100
Median
100
100
100
Range
100-100
20-100
100-100
SD
0.00
35.78
0.00
Participant 1
0
0
1
Participant 2
0
0
1
Participant 3
2
3
0
Participant 4
1
0
0
Participant 5
1
1
0
Error Rating
Mean
0.8
0.8
0.4
Median
1
0
0
Range
0-2
0-3
0-1
Standard Deviation
0.84
1.30
0.55
Assisted Completion Rate
0/5
0/5
0/5
Unassisted Completion Rate
5/5
4/5
5/5
Completion Rate
5/5
4/5
5/5
Recommendations
Findings
List your findings – positive / negative for every task and give an explanation
Observations
//List your observation
Recommendations:
//List the recommendation and include the rationale behind it
Overall Conclusion
// Conclude with an overall idea about the website, issues and recommendations
References
Dream in code: Available on: http://www.dreamincode.net. Accessed on 28th April 2015.

HCI in the software process. Retrieved from: http://www.hcibook.com/e3-docs/slides/notes-pdf/e3-chap-06-6up.pdf. Accessed on 28th April 2015.
Appendix A – Data Collection Forms

Screening Questionnaire
Orientation Script
Pre Test Questionnaire
Video Consent Release Form
Task Scenarios
Post Test Questionnaire
Data Logger
Facilitators Checklist
Appendix B – Screen Shots
Embed screenshots of website and highlight the usability issues/ observation Read More