Evaluative UX research: Examples and methods for high-impact experiences

Evaluative research is a research method used to evaluate a product or concept and collect data to help improve your solution. It offers many benefits, including identifying whether a product works as intended and uncovering areas for improvement.

Also known as evaluation research or program evaluation, this kind of research is typically introduced in the early phases of the design process to test existing or new solutions. It continues to be employed in an iterative way until the product becomes ‘final’. “With evaluation research, we’re making sure the value is there so that effort and resources aren’t wasted,” explains Nannearl LeKesia Brown, UX Researcher at Figma..

According to Mithila Fox, Director of Product Research at Contentful, the evaluation research process includes various activities, like content testing, assessing accessibility or desirability. During UX research, evaluation can also be conducted on competitor products to understand what solutions work well in the current market before you start building your own.

“Even before you have your own mockups, you can start by testing competitors or similar products,” says Mithila. “There’s a lot we can learn from what is and isn't working about other products in the market.”

Evaluative research is often conflated with evaluative research design, but they’re not the same. The key difference between the two lies in their scope and focus.

Evaluative research focuses on assessing the usability, effectiveness, and performance of a product. Common methods used in evaluative research include usability testing, UX surveys, A/B testing, tree testing, and heuristic evaluations.

Evaluative research design is a strategic framework that guides how evaluative research is conducted. Evaluative research design is used in various fields, including education, social sciences, and healthcare.

Evaluative research design includes three main types of evaluations: formative, summative, and outcome (more on these shortly).

Evaluative research is crucial in UX design and research, providing insights to enhance user experiences, identify usability issues, and inform iterative design improvements. It helps you:

Evaluative research is an ongoing process throughout the product design and development cycle, from early design stages to post-launch. Here are some of the key times to conduct evaluative research:

The difference between generative research and evaluative research lies in their focus: generative methods investigate user needs for new solutions, while evaluative research assesses and validates existing designs for improvements.

Generative and evaluative research are both valuable decision-making tools in the arsenal of a researcher. They should be similarly employed throughout the product development process as they both help you get the evidence you need.

Generative research methods include:

Evaluative research methods include:

Some research methods, such as UX surveys and user interviews, can be used in both generative and evaluative research, depending on the context and goals.

When creating the research plan, study the competitive landscape, target audience, needs of the people you’re building for, and any existing solutions. Depending on what you need to find out, you’ll be able to determine if you should run generative or evaluative research.

Mithila explains the benefits of using both research methodologies: “Generative research helps us deeply understand our users and learn their needs, wants, and challenges. On the other hand, evaluative research helps us test whether the solutions we've come up with address those needs, wants, and challenges.”

There are three types of evaluative studies you can tap into: summative research, formative research, and outcome research. Although summative evaluations are often quantitative, they can also be part of qualitative research.

TL;DR: Run formative research to test and evaluate solutions during the design process, and conduct a summative evaluation at the end to evaluate the final product. Measure specific outcomes through outcome evaluations to understand the impact on user behavior and satisfaction.

Formative research is conducted early and often during the design process, to test and improve a solution before arriving at the final design. Running a formative evaluation allows you to test and identify issues in the solutions as you’re creating them, and improve them based on user feedback.

A summative evaluation helps you understand how a design performs overall. It’s usually done at the end of the design process to evaluate its usability or detect overlooked issues. You can also use a summative evaluation to benchmark your new solution against a prior one, or that of a competitor’s, and understand if the final product needs assessment. Summative evaluation can be used for outcome-focused evaluation to assess impact and effectiveness for specific outcomes—for example, how design influences conversion.

Outcome evaluation research assesses the effectiveness of a design by measuring the changes it brings about in specific user behaviors and satisfaction. This type of evaluation focuses on the short-term and long-term impacts on users, such as improved task completion rates or increased user engagement. Outcome evaluations help understand the real-world effectiveness of a product design and support data-driven decisions for future improvements.

“Evaluation research can start as soon as you understand your user’s needs,” says Mithila. Here are five typical UX research methods to include in your evaluation research process:

User surveys can provide valuable quantitative insights into user preferences, satisfaction levels, and attitudes toward a design or product. By gathering a large amount of data efficiently, surveys can identify trends, patterns, and user demographics to make informed decisions and prioritize design improvements.

There are many types of surveys to choose from, including:

Unlike open or hybrid card sorting, closed card sorting uses predefined categories to evaluate specific usability issues in an existing design.

By analyzing how participants group and categorize information, researchers can identify potential issues, inconsistencies, or gaps in the design's information architecture, leading to improved navigation and findability. The results offer quantitative insights into users' mental models and expectations.

Closed card sorting allows researchers to:

Tree testing, also known as reverse card sorting, is a research method used to evaluate the findability and effectiveness of information architecture. Participants are given a text-based representation of the website's navigation structure (without visual design elements) and are asked to locate specific items or perform specific tasks by navigating through the tree structure. This method helps identify potential issues such as confusing labels, unclear hierarchy, or navigation paths that hinder users' ability to find information.

Tree testing allows researchers to:

When comparing tree testing vs. card sorting, tree testing evaluates the effectiveness of a site's navigation by having users find specific items in a text-based hierarchy. On the other hand, card sorting involves organizing information into categories to reveal users' mental models and improve information architecture.

Usability testing involves observing and collecting qualitative and/or quantitative data on how users interact with a design or product. Participants are given specific tasks to perform while their interactions, feedback, and difficulties are recorded. This approach helps identify usability issues, areas of confusion, or pain points in the user experience.

There are different types of usability testing, including:

A/B testing, also known as split testing, is an evaluative research approach that involves comparing two or more versions of a design or feature to determine which one performs better in achieving a specific objective. Users are randomly assigned to different variants, and their interactions, behavior, or conversion rates are measured and analyzed. A/B testing allows researchers to make data-driven decisions by quantitatively assessing the impact of design changes on user behavior, engagement, or conversion metrics.

To gather valuable data and make better design decisions, you need to ask the right research questions. Here are some examples of evaluative research questions:

Usability questions

Get inspired by real-life usability test examples and discover more usability testing questions in our guide to usability testing.

Product survey questions

Discover more examples of product survey questions in our article on product surveys.

Closed card sorting questions

Find out more in our complete card sorting guide.

Across UX design, research, and product testing, evaluative research can take several forms. Here are some ways you can conduct evaluative research:

This example of evaluative research involves conducting usability tests with participants to compare the performance and user satisfaction of two or more competing design variations or prototypes.

You’ll gather qualitative and quantitative data on task completion rates, errors, user preferences, and feedback to identify the most effective design option. You can then use the insights gained from comparative usability testing to inform design decisions and prioritize improvements based on user-centered feedback.

Cognitive walkthroughs assess the usability and effectiveness of a design from a user's perspective.

You’ll create evaluators to identify potential points of confusion, decision-making challenges, or errors. You can then gather insights on user expectations, mental models, and information processing to improve the clarity and intuitiveness of the design.

Conducting diary studies gives you insights into users' experiences and behaviors over an extended period of time.

You provide participants with diaries or digital tools to record their interactions, thoughts, frustrations, and successes related to a product or service. You can then analyze the collected data to identify usage patterns, uncover pain points, and understand the factors influencing the user experience.

In the next chapters, we'll learn more about quantitative and qualitative research, as well as the most common UX research methods. We’ll also share some practical applications of how UX researchers use these methods to conduct effective research.