Chapter 2

Moderated vs. unmoderated usability testing: Which one is right for your product build?

There's no right or wrong decision between moderated and unmoderated usability testing. Both UX research methods are highly successful in gaining crucial user experience insights to help you build more user-centric products.

However, both moderated and unmoderated usability testing methods come with their own pros and cons. By the end of this read, you'll be able to identify which method is best for your product team, workflow, and budget.

What is moderated usability testing?

Moderated usability testing is human-to-human, real-time testing. For this type of testing, a moderator (also called the facilitator) guides participants through the test, and is available throughout.

A moderated usability test enables moderators to record observations, ask or answer questions, and ask follow-up questions.

For example, during moderated user testing, a facilitator can deep-dive into unplanned topics that emerge in-the-moment, and follow organic conversational threads to uncover unexpected insights. Whereas unmoderated user testing only allows testers to answer the questions your product team provides in advance (although you can follow-up retrospectively).

Today, moderated tests can be done remotely or in person, as long as the moderator is able to read participants’ body language and facial expressions.

Moderated user testing has been thought to limit the reach of diverse test participants, as product teams needed to rely on physical proximity or travel to implement their tests. The other limitation is that post-test analysis can be laborious—requiring researchers to manually shift through recorded interviews to find results.

However, today, video conferencing tools like Google and Zoom allow researchers to connect with participants around the world. On the analysis front, remote UX research tools are using AI to advance and streamline moderated test analysis.

For example, Maze’s AI-powered Interview Studies enables researchers to mass-analyze moderated interview recordings, enabling teams to quantify qualitative feedback, summarize interviews, and surface critical insights faster than ever before.

Pros of moderated usability testing

A moderated session’s benefits include:

  • Allowing the moderator to read body language and facial expressions
  • Helping the moderator understand how engaged someone is with the product
  • Enabling participants and moderators to establish trust and gather more candid feedback
  • Providing an opportunity for the moderator to ask follow-up questions
  • Allowing the moderator to dive into emerging user pain points
  • A reactive collection of additional information on lo-fi prototypes

Cons of moderated usability testing

Some of the downsides of moderated usability tests are:

  • They can be more time-consuming when it comes to organizing and aligning calendars
  • Test results take a while to sort through and dissect (although this can be made easier with tools like Maze)
  • Difficulty in coordinating schedules may limit your target audience reach to those within a similar time zone
  • They can come with additional costs, as product teams may need to rent a quiet testing space and cover travel expenses
  • They can be prone to cognitive biases (from participant or moderator), such as social desirability bias
  • They’re often limited to focus on the start of a product build, due to the amount of organization needed and available resources further into product development

4 Tips for more successful moderated usability tests

Maze’s Product Researcher, Matthieu Dixte, shares four ways product teams can run smoother moderated user tests from the beginning.

1. Stick to the predefined questions in your test script

Changing just one word can influence a user's interpretation of a question or task instruction. Therefore, it’s important to formulate the questions in the same way for each respondent to ensure you’re comparing the same behaviors and learnings.

Keep your initial questions identical between participants—if follow-up questions emerge, consider asking these across the board, too.

2. Listen, and don’t be afraid of silence

During the natural flow of a test, you should adopt a neutral and silent posture. It’s important to let the person complete their task, to search, and struggle—that's where you’ll gain the most valuable insights.

Matthieu explains: "Don’t react to the actions performed. Whether the person easily completes a task or struggles with it, do not show any hints or emotions."

Instead, be curious and eager to understand their behavior: “Observe and give space to the participant. If you help them by giving hints, it's like taking the mouse for them and completing the task yourself.

“If the person feels uncomfortable, remind them that there are no right or wrong answers, no pressure, and that you’re not judging them.”

3. Stay curious

When asking follow-up questions, be cautious about cognitive biases that might slip in. As with any best practice in exploratory interviews, it’s important to remain neutral and avoid leading questions.

Instead, promote rephrasing and introspection on the respondent's side, rather than suggesting your interpretations. This means letting them frame and vocalize the question on their own, rather than interpreting their struggles and framing their answer for them.

4. Be conscious of the unconscious

Moderated user interviews can be heavily subjective when noting behavioral trends around body language and facial expressions. What one moderator considers worth noting down, another may not even notice.

What’s more, two individuals’ expressions of the same emotion can differ drastically—be that due to cultural differences, different neurotypes or ages. This impacts both how a participant expresses an opinion, and how a moderator interprets it.

It can be a challenge to avoid accidentally influencing, misinterpreting, or creating bias in the test. Work to create a thorough testing guide (for both moderators and participants) that covers best practices, UX research ethics, and ways to minimize the impact of cognitive biases.

When to use moderated usability testing

Moderated usability tests lend themselves to a time in the product development process when more explanation and guidance may be needed, from both the researcher about the prototype, and the participant about their answers. For example, during wireframe testing, when a candidate may need live support to complete tasks, or the limited functionality of the prototype may need further explanation.

This means moderated testing naturally slots into the beginning of a design process—when qualitative insights about opinions and overall feel of product can help guide the later functionality and specific UX.

Using moderated testing early on gives UX teams time to return to the participant throughout the build, dive deeper on their responses, and show them renditions after feedback.

This promotes continuous product discovery, providing design teams with feedback from ideal customer profiles and allowing them to check in with participants to see if they’ve delivered on feedback in each product rendition.

Moderated testing can also help UX teams:

  • Build out product use cases for stakeholders: Helping to win buy-in for further budget and resources
  • Run customer-centric competitive product analysis: To better understand processes and platforms customers are using before they find your tool
  • Gauge usability in early-stage prototypes: Even with lo-fi prototypes, moderated tests can help UX designers understand if they’ve ordered information correctly
  • Unpick customer journeys: Identify where customers are at before entering the product, where they’re most likely to go first, and why

What is unmoderated usability testing?

Unmoderated usability testing is, unsurprisingly, when the test doesn’t involve a moderator. Users complete the task independently, often using usability testing tools that record their decisions and actions.

This type of test process is self-paced and the test taker is left to complete the tasks and answer questions to the best of their ability—they cannot ask for help. Testers are usually in their own space.

Unmoderated, remote usability testing is commonly used for high-fidelity prototypes that need little-to-no explanations. For example, in the final stages of design when you have a production-ready, interactive app that’s lacking the final tweaks before launch.

Unmoderated usability testing is also used when product teams:

  • Need access to a lot of responses quickly
  • Want to engage with broader demographics
  • Are looking to collect quantitative data to prove or disprove theories
  • Have a tighter budget

Pros of unmoderated usability testing

The pros of unmoderated usability testing methods include:

  • Allowing for a diverse and distributed group of testers
  • More accessibile for participants with accessibility needs
  • Flexibility to weave into a product build workflow
  • Lower costs (less spend on human resources and space rental)
  • More hands-off, allowing smaller product teams to conduct more research
  • Quicker to implement, especially when using a research tool
  • Can gather vast quantities of data in a short period of time
  • Fast validation of designs and concepts, aiding business cases with the numbers
  • Participants can complete tests in their own time while teams continue working on other priorities
  • Great for observing trackable behavioral patterns of a user in their natural environment

Cons of unmoderated usability testing

Some of the cons to using unmoderated testing sessions are:

  • It’s difficult to track off-screen user behavior
  • Gathering sentiment may be harder
  • Due to a lack of moderator, there may be more user drop-off or unfinished tests
  • A participant may struggle to voice frustrations
  • Users can be easily distracted

An unmoderated user test also has to be air-tight, leaving no room for questions. This could mean multiple renditions until testers are totally clear on what’s expected of them.

Research tool features like Maze Clips are combating some traditional cons of unmoderated usability testing by enabling researchers to capture short audio, video, and screen recordings from asynchronous prototype and website tests.

This means Clips can help to bring some moderated user testing benefits to unmoderated remote tests, such as enabling product teams to gather qualitative behavioral insights or understand unclear responses and slow interactions by tracking their activities.

When to use unmoderated usability testing

Unmoderated usability tests are incredibly valuable at any point in the design and development process. They’re particularly useful when you need a large amount of data, or want to hear from diverse participants.

Unmoderated usability testing is also best used if your team is on a tight budget, or has limited time and resources to dedicate to the task—as you can set up a usability test and leave it to capture responses in the background, while you focus on the next task at hand.

It’s a good fit for any straightforward UX processes where no handholding is required, so is often used further along the design process when you have high-fidelity prototypes to test.

Unmoderated usability tests can also be used to help design teams:

  • Assess navigation: Run tree testing to see if users can navigate from one section of the design to another in a way that feels logical to them
  • A/B test prototypes: Split test everything from graphics to button colors and in-app messaging with an A/B test for usability
  • Test hypotheses: Set up an unmoderated usability test to see if what you think will work actually does, then move forward based on data
  • Product messaging tests: Supporting the launch of your design and helping your team to decide on which copy to use in the product and marketing

Despite unmoderated usability testing typically coming towards the end of a design process when reviewing final designs or testing prototypes, when participants are briefed well and given an intuitive testing platform, many organizations opt for unmoderated usability tests throughout the design process as a low-cost, high-reward option.

Moderated vs. unmoderated usability testing: Which is best, and how to choose between them

There’s no right or wrong answer as to whether you should use moderated or unmoderated usability tests. Both moderated and unmoderated usability testing methods are valuable in any design process.

Here are some questions to keep in mind when deciding between the two:

  • What budget do you have available?
  • What usability metrics are you looking to track?
  • How quickly do you need results?
  • What stage are you at in your design: high-fidelity or low-fidelity?
  • Where are your target audience located?
  • What knowledge gap do you need filling to continue your design?

Once you’ve got these questions figured out, you’ll be in a much better position to decide which usability testing type you need. In some cases, you may benefit from using both, as Matthieu recommends:

“I use a mix of the two methods. It's not necessarily one or the other in confrontation, but a combination of the two approaches, which are complementary to take advantage of each other.”

Thankfully, Maze has you covered no matter which method (or mix) you choose. With Maze, you can use AI-supported Interview Studies to streamline and manage moderated tests, or opt for remote unmoderated usability tests to gather high-quality data at pace.

Either way, your team can download customizable research reports, collaborate with team members, and access data that helps you understand user behavior in context.

Both solutions eradicate fears and concerns product teams typically have about each, while empowering them to run both moderated and unmoderated user tests harmoniously.

In the next chapter, we’ll cover the steps to conduct live website usability testing, alongside insights from industry experts, so you can apply the learnings to your process.

Enjoy the best of both worlds

Save time worrying about multiple tools and consolidate your moderated and unmoderated tech stack into a single, holistic research tool.

Frequently asked questions about moderated vs. unmoderated usability testing

What’s the difference between moderated and unmoderated usability testing?

Moderated usability tests need a facilitator to host, support, and collect responses whereas unmoderated tests see the participant complete tasks on their own. Both provide essential usability insights in product design and development.

What are the advantages of unmoderated usability testing?

Unmoderated usability tests are cost-effective, rapid, and provide quantitative data to build business cases and prove or disprove theories. They enable UX researchers to access a vast and diverse candidate base.

What are the benefits of moderated usability testing?

A moderated testing process is held on-site with a facilitator. This enables conversation between the candidate and the facilitator, providing in-depth insights, real-time customer feedback, and an opportunity to read user body language.

How to choose between moderated vs. unmoderated usability testing?

To pick between the two, your product team will need to consider the budget, turnaround time, project development stage, desired ICPs, and the knowledge gap you’re looking to fill as the two methods will provide different insights.