Ensuring validity in research: Key methods, challenges, and solutions

How can you be sure your UX research insights reflect real user behaviors, motivations, and habits?

Ultimately, it’s a question of ensuring your study accurately measures what it’s supposed to measure—aka research validity.

By strengthening validity in UX research, you can ensure that your findings are credible and trustworthy. Designing research projects with validity in mind ensures you get valuable, usable insights.

In this article, we show you how to ensure you collect valid findings for your next research study, along with solutions to the common validity challenges you may encounter along the way.

Validity in user research is the extent to which your UX research study accurately captures and reflects user behavior through specific methods and measurements. A valid study is credible and trustworthy. Ensuring validity in your research study leads to actionable user insights that align with your research objectives.

To ensure research validity, you’ll want to look at:

Research validity is a common counterpart of reliability, which looks at how consistently you can replicate your data collection method and get the same reliable results.

So why pay special attention to validity before embarking on an ambitious research project? The main benefits include:

Research validity can be put into two broad categories: internal and external validity.

Internal validity measures the level at which you derive results from the specific variables you’re testing, rather than external factors (also called confounding variables). It answers the question: “Did what I tested truly cause the result?”

Internal validity is:

External validity looks at how accurate your study would be beyond your sample size, with a focus on generalization in other settings, populations, and measures. It answers the question: “Do my findings apply to the real world?”

External validity is:

How do these two main validity types play out in UX research?

There are other important validity types as well:

Ensuring validity in research doesn’t mean you need to overhaul your entire UX research study. It just means taking a few extra steps to ensure your insights are credible and trustworthy.

Here are five methods to help you strengthen your study’s validity.

Always ask neutral questions when conducting a survey questionnaire, usability test, or interview study. Your questions shouldn’t lead participants into answering in a specific way.

For example, the question, “Wasn’t that easy?” can influence participants to affirm, skewing results.

Instead, to mitigate bias, ask an objective, non-leading question like, “How easy or difficult did you find completing the task?”

Sticking to an objective line of questioning strengthens internal validity by reducing the risk of influencing participant responses.

You may have your test all rehearsed and ready for launch. But that doesn’t guarantee a flawless session or valid results. Before conducting the real test, try it out with other members on your team. This can quickly uncover confusing tasks, unclear instructions, and inconsistent procedures early on.

For example, a pilot test might reveal that your interview questions are biased or that your task flow is too complex.

Run a pilot test study when you want to check for internal validity and limit mistakes that could skew your results.

If you want your study to give you a high-resolution snapshot of user behaviors, consider testing the same set of participants with different qualitative and quantitative research methods.

For example, if you conduct a mixed research method project with user interviews followed by closed-ended surveys, you gain a combination of qualitative and quantitative insights.

If you get the same or similar results across different types of research methods, it indicates that your study is valid across a wide set of users and has statistical significance.

Using mixed research methods supports construct validity because it cross-references results via different methods. It also strengthens external validity because it demonstrates that you can get consistent results that hold up in both quantitative and qualitative research.

While your participants should share the behaviors of your user persona, be sure to recruit testers with different demographics, devices, and geographic locations.

Recruiting a diverse participant sample ensures that your research is inclusive and your end product is accessible and helpful for ensuring external validity. Recruiting a diverse user base and getting the same results proves your findings are generalizable across use cases.

When conducting tests with users, make sure that the tasks and questions you give them directly relate to their goals and jobs-to-be-done. This ensures that the insights you receive are relevant to your study, and will be helpful when making design decisions.

For example, if you’re exploring how users interact with your budgeting app, ask them about the goals they set, how they fulfill them, and any other apps they use alongside yours. If you’re testing usability, frame tasks around your users’ needs, like “send a payment request and get paid.”

Connecting your testing questions to your users’ jobs-to-be-done gives you additional context and a better, wider, and deeper understanding of your user base, which strengthens content validity.

Ensuring validity in your UX research studies requires a proactive approach to potential challenges. Here are three roadblocks standing in the way of valid insights in your next UX research study.

Faced with tight deadlines, budgets, and resources, many researchers recruit the first testers they come across instead of finding people who match their real-world user base.

Failing or rushing the process of finding and vetting the right research participants is a huge mistake; when you recruit the wrong testers, your user insights can be false or skewed, which raises questions about your research validity.

One way to avoid this problem is to use a recruitment tool like Maze Panel, which gives you access to over 3 million vetted testers across 130+ countries.

Consider also using stratified sampling to find a diverse set of participants. This sampling technique involves dividing potential participants into subgroups, or “strata,” based on demographic information such as age, education, income, or employment. You then choose participants from each stratum at random to participate in your test. This ensures that your user base is diverse, providing you with valid insights.

Bias is when personal leanings, yours or the participants’, skew your study’s data and, in turn, the insights you draw from it.

When bias creeps in, your feedback becomes unreliable, derailing your path to relevant findings.

Types of bias in UX research to be aware of:

User research gives you a wealth of user data and insights that you can apply to product development and design. But drawing conclusions from a single data source can jeopardize your study’s validity; participants might say one thing in interviews, but behave differently during a usability test.

The best way to overcome this challenge is to use mixed research methods, and introduce data triangulation into your UX workflow. Data triangulation combines multiple data types, sources, and methods in order to strengthen validity. For example, interviews could work in hand with prototype testing, while In-Product Prompts can be coupled with heatmaps.

Once you have results from each research method, cross-check your findings for accuracy. If you’re conducting usability tests, be sure to include multiple metrics—like task success rate or time on task. If you’re running a set of surveys or interviews, ask a wide range of question types, including open-ended, closed-ended, and multiple-choice, and use Likert scales.

Research validity shows how accurately your research reflects real user insights. By ensuring validity through methods like asking objective questions, pilot testing, and using mixed-method research, you can rest assured that your design decisions are rooted in real user behaviors. Besides credible insights, paying attention to validity means you can create a connection between your methods, measurements, and original research goals.

To ensure your insights are valid, we recommend using a tool like Maze. From running pilot studies with your team to conducting mixed-methods research, you can use Maze to streamline valid, reliable insights for your next project.