Types of Testing

December 15th, 2011, Posted by Mark Sloan

Testing is something that should continue throughout a project. From the very beginning, testing can mean something as simple as putting sketches in front of coworkers, friends, and family to formalized recruitment of actual users with a testing facility. The key is to consistently get feedback, both formal and informal to test your designs (hypothesis) and adjust accordingly. Analyzing the feedback appropriately is often as tricky as setting up the proper testing method in the first place.

“When to Use Which User Experience Research Methods” is a great article on Jakob Nielsen’s Alertbox by Christian Rohrer. It summarizes a variety of research methods into three dimensions and then creates a graph for mapping them out against one another. Even here, the “How to fix” part is not straightforward and often requires multiple designs to determine which is most successful. As seen in Our Methodology, these different tools help us measure SEED (satisfaction, efficiency, effectiveness, desirability). Depending on the project goals, budget and timing, user testing may be of types Exploration, Assessment, or Validation.

Exploration: Focus on learning more about the users and their thought processes. May be in person with a facilitator, remote, or even through use of surveys. Mostly qualitative in nature. Is useful for comparing alternative designs and branding initiatives and may include focus group testing.

Assessment: Determine whether designs are performing adequately, at a level of completion of tasks and general usability of the site. This is typically performed with a prototype that allows users to interact with the system directly, rather than through a facilitator. Captures both qualitative and quantitative data.

Validation: Without preset goals or a baseline, it is difficult to measure improvements. For validation to get the best data, comparing previous design or expectations to new results is most valuable. Ability to measure time-on-task, errors, efficiency and learnability can prove or disprove design patterns.

Testing Methods

The methods and tools that are best for any testing are largely influenced by the testing goals and budget. Three types of methods allow for flexibility in achieving those goals.

Online Survey/Tool: Survey forms provide user self assessment feedback and click/task tracking tools are objective measures of success. Because this can often be implemented via site intercepts, recruitment costs are minimal. Tools can range from $ to $$$$ depending on features.

Online Facilitation: An extension of the online tools, these allow a facilitator to investigate user reactions and/or questions in real time. Often this requires users to have certain software (such as Flash) installed and rarely can facilitator see user body language. Again, this can utilize conference call type software to specialized software ranging from $$ (facilitator time) to $$$$.

In Person Facilitation: This includes one on one and focus group types of testing. Allows facilitator to adapt in real time to users and capture more qualitative information. This often requires recruitment fees as well as facility and travel costs on top of software. Cost $$$ to $$$$.