The following is an article first published in March 2000 on the Puget Sound Research Forum's website. It was an assesment of website usability testing at that time. Much of it still applies today.

Web Site Usability Testing: What You Should Know

by Nancy Hardwick

Background
Usability testing of Web sites evolved from usability testing of tools and hardware for ergonomics, software, ATMs, automated phone systems, etc. As stated on Sun.com, usability testing is now applied to Web sites to test utility, learnability, efficiency, retainability, errors and satisfaction.

Usability testing can result in changes to the way a site is navigated, its screen design and layout and the terminology used on the site. It can also be used to optimize consistency of design and terms, and to make the site more intuitive for the user.

Methodology
Generally, a small sample size is adequate for usability testing. According to Jakob Nielsen, the usability guru, five respondents are enough to identify major improvements that need to be made to a site. Additional respondents will only identify minor concerns that will cloud the issue of which improvements should be addressed first. It is recommended that additional testing be conducted after the first round of improvements have been made to identify more work to be done.

In some cases, more than five respondents may be needed. If there are multiple target audiences for your website, it may be a good idea to gather five respondents from each group if they are disparate enough. For example if your site is expected to attract young homemakers and senior citizens, it would be wise to test among both groups.

We typically recommend conducting a minimum of 12 interviews. Although Jacob Nielsen suggests fewer, our experience shows that 12 usability tests provide a better sample. The additional tests allow us to uncover a variety of issues that we might not have found using just five. Five subjects could work if one were just looking for a general idea of the issues, while 12 really enables you to identify navigation problems and determine the necessary changes.

There is some debate about whether to conduct usability testing with subjects individually or in pairs. Using a computer is by nature an individual activity, so conducting the tests with one person at a time is most representative of the actual user experience. Additionally, there is no opportunity for multiple subjects to give each other clues about how to complete an exercise, which is important if time and number of clicks to complete an exercise are being tracked. Time to complete an exercise cannot be accurately tracked in paired testing, as time is not solely spent on navigating through the exercises, but also on discussion. There can be some benefit to testing subjects in pairs, however. Users can be observed discussing their thought processes as they complete the exercises. Sometimes they are more forthright with each other than they would be with a facilitator.

The usability study can be conducted using paper mock-ups of the screens, or on PCs. Paper mock-ups can be a quick and easy way to address questions relating to the way the site content is presented graphically and linguistically. Multiple mock-ups can be created to get feedback on different approaches to wording or design. This method is more cost effective than preparing the site for testing, and is better for use early in the design process. While mock-ups are good for testing the visual aspects of a site, conducting testing on PCs is optimal for addressing navigation issues later in the design process.

The test is set up so that each subject is asked to complete a number of tasks on the site. For an e-commerce site, subjects can be asked to find a given product down to the SKU level, and complete the purchasing process. They can also be asked to change the number of items they want or add/change the delivery address. For a service site, participants can be asked to use certain services offered by the site. The tasks assigned should increase in complexity somewhat as the subjects acclimate to the site layout.

Time should be allowed after the exercises are completed for the facilitator to ask questions about the subject's experience with the site. There can be some questions that are asked of each respondent, as well as questions customized based on what was observed. For example, if a participant doesn't make use of the site's search feature while looking for a product specified in the exercise, the facilitator can probe for reasons why.

Usability testing can run up to two hours in length for each subject, from start to finish. Videotaping of the test is recommended for later viewing. This makes it easy for engineers to go back and review problems identified, and also allows for other team members to see what happened during the test. Typically, the video camera is set up to record the PC screen and the subject's face. This way, movement of the mouse can be observed as well as facial expressions.

Summary
Web site usability testing has become a key part of marketing programs, enabling developers and marketers to learn about customer's experiences while visiting the Web site. Its ability to provide feedback on site design, functionality and intuitiveness from the point of view of the customer is invaluable.

Sources for More Information

Web sites:

Books:

A Practical Guide to Usability Testing by Joseph S. Dumas and Janice C. Redish

Other Related Topics:
Usability testing
Strengths and weaknesses of usability testing