The 6 steps of concept testing: How to run a product concept test

A product concept test validates the solution you and your team have come up with before you pour resources into the idea. “It should be one of the first steps in the product discovery phase,” explains Shelly Shmurack, Product Manager at Walmart Global Tech.

A concept test gauges your target audience’s reaction to your proposed solution, which helps confirm whether or not your product adds value for your users.

There are a few ways to conduct a concept test, but the main two methods are moderated and unmoderated testing. Neither method is better than the other, but the one you'll choose depends on your goal and the stage of the process you're in.

Unmoderated testing, usually in the form of surveys, is handy near the start of the product development process for a general idea of customer preferences. It’s a quick and affordable early idea validation method.

On the other hand, moderated testing, such as user interviews or focus groups, can be used for deeper analysis later in the process when you have a minimum viable product (MVP) or prototype to showcase users.

Effective planning will help ensure your product concept test provides you with useful insights. Here, we’ll look at the steps you’ll go through when setting out to conduct an unmoderated product concept test. For moderated testing, these steps might differ slightly but are still useful to understand.

Outlining what exactly you’re trying to learn from your concept test will help guide which type of test you use and the questions you employ. Setting a goal will help ensure focus in your test, rather than a scattershot attempt to see if your customers merely “like” the concept at hand.

As Grainne Conefrey, Product Manager at NewsWhip, puts it, deciding how to conduct their concept test “depends on the goal we have set with the team and the information we’re trying to understand.”

The script is really the core of the concept testing, so getting it right is vital. Start the script with an introduction to your concept and the questions you ask your target audience.

Your questions can capture participants’ first reactions to the concept, their specific likes and dislikes, how they’d use the concept in real-life situations, and their likelihood to purchase the product. Here are a few example questions:

Be careful about the language you use. Avoid leading questions like “What makes Concept A better than Concept B?” as that question shows an innate preference for one concept over the other, and introduces bias into your results. Biased answers will invalidate your product concept test, wasting both time and money.

There don’t need to be too many questions, though, as it could lessen your overall learning:

It goes without saying that a product concept test is looking for insights from your target market, so you need to recruit people that fit your audience’s defined characteristics.

That said, within your target audience you should try to keep some variation with regards to age, gender, and other factors so that you can see which features of your concept appeal to specific subsections of your target audience. Just always keep in mind who your overall target is regarding behavioral patterns, demographics, and consumer traits.

And don’t forget to thank them for taking the time to complete the test. You can also add monetary compensation, like a gift card, to reward participants.

Determining the flow of the test will allow you to create a survey that makes sense to users in an unmoderated setting. Start with an introduction to the concept. You can do this with a prototype or mockup of your concept and a short explanation of what the concept is.

Next, have a couple introductory demographic questions. This will assist you in your later analysis, seeing which parts of the concept appeal to which customer segments most. Figuring out the participants’ buying history can also be helpful in seeing where your concept fits in the bigger picture. Ask about their awareness of your brand, what brands in your category they’ve bought in recent months, and how frequently they purchase or consume these items.

Now down to questions about the concept overall. You can start with general questions about the solution, then dive into the specific characteristics of the product concept. Comparisons and buying response questions are a useful way to round out your test.

At the end of a product concept test, you’ll have a lot of insights and data. However, it may be hard to analyze all of this information at once, especially if most of your feedback is qualitative. Adding in some quantitative elements will help make your results more digestible.

There are a few simple ways to do this. You could assign point values to certain responses or sort participants’ responses based on what percentage of the test was completed.

Top-box scores are especially helpful for making survey data more accessible. Top box scores are what they sound like: they’re the top scores available on a survey.

For example, in a five-point Likert scale with 5 being the most positive score, a 4 or a 5 would be deemed a top-box response. You and your team can use top-box scores when analyzing your survey data to deem what percentage of respondents gave top-box scores to the concept or certain parts of the concept.

With a quantitative component, it’ll be easier for you to rank concepts against each other and find the idea that’ll work best for your target audience.

Once you’ve got your results through, it’s time to get analyzing. In some senses, the hardest part of concept testing–that is, running the test itself–is over, but the journey is also just beginning…

Untangling the results data from your concept test may feel overwhelming, but it’s imperative to organize the results and feedback, not only for yourself, but for any stakeholders involved. If you use a user testing tool like Maze, you may find this has built-in capabilities to turn the results into easily-digestible reports.

Using a tool to initially sort through the raw results and filter the data makes the challenge of this step considerably more manageable, and enables you to focus on what you do best: interpreting the results in context of product and design decisions. Equipped with the takeaways of your concept test, you can now proceed to review feedback and move forward with the right ideas.

Mistakes may happen during your concept test, despite all of your planning. But it’s not the end of the world.

To help start you off on the right foot, though, we have some common concept testing mistakes you may encounter and just how you can avoid them.

Always keep your target audience in mind when selecting respondents for your product concept test. It’s understandable to want to get as much information from as many people as possible, but casting your net too wide and selecting the wrong respondents can backfire. “The wrong testers will mainly cause you to waste money and time,” Shelly says.

To keep from making this mistake, outline clearly the parameters of who falls into your target audience and use these to select your participants. Market research and drafting user personas can help you determine who exactly makes up your target audience.

While you’ll likely use that information when selecting your participants, you can also include some questions about demographics in your survey to ensure that your respondents are a part of your target audience.

Consider what stage you’re at in your product discovery journey when deciding on your testing method. A concept testing survey is most helpful when you need quick feedback from lots of participants. This is most relevant at the early stages of product discovery when you’re just gathering information about users’ pain points and needs. Or, similarly, you can run a survey at the end of the process to gather statistically relevant, quantitative data on a particular concept before moving into the development stage.

In between these two stages, when you need to dive into slight variations of a concept and really understand how your users feel about a product idea, more in-depth focus groups or user interviews can provide the qualitative information you need to really get to the why behind the choice.

If you have too much going on in your product concept test, you’re not going to get the streamlined insight you need. Your data will end up all over the place, and participants will end up confused and tired, which leads to a reduced completion rate.

To avoid participants getting confused by your questions, always test your test. “Start with a few minor iterations until you are sure the testers understand your test questions correctly,” Shelly explains.

Netali Jakubovitz, Senior Product Manager at Maze, says that trialing the test with your team can help you figure out potential friction points in your test.

“Run your test with an internal team to make sure the questions are well understood, the survey is not overwhelming or tiring, and that the way you collect answers to specific questions would allow you to process them and inform decision making later on.

For example, would open-ended questions enable data processing? Would multi-select questions generate confusion instead of contributing to focus? Do you really want to enable that ‘other’ option? All of these are valid but should be done intentionally,” Netali recommends.

When coming up with a solution, you surround yourself with information about the problem at hand. You and your team have probably spent most of your time thinking about your concept and every detail about it.

However, your concept test participants are missing this context. They’re just people with a problem that your idea will hopefully solve. Remember that they aren’t experts and eliminate jargon from your questions and test. Like above, testing your test internally before you share it with respondents will help spotlight any language that needs tweaking.

Hearing that your concept doesn’t resonate with your target audience can feel discouraging. That’s not a sign to give up, though.

Getting less-than-perfect results means you need to analyze the results to understand why the idea isn’t working. Perhaps you’re testing with the wrong audience, you’ve made some assumptions about the product that aren’t holding in real-life, or it just needs a bit of tweaking to meet users’ requirements.

Whatever the reason, make sure you understand why the concept isn’t working and work on it until you get it right. The whole reason you’re testing at this stage is to avoid massive reworks later in the process, so the fact that the results aren’t satisfactory is a good thing. That means you’re doing your job—just keep at it.

On the other hand, maybe you get back great results. Your target audience validates your concept, and you feel ready to move forward with the rest of the product development process.

However, don’t stop there. Conducting multiple product concept tests during the entire product development process will yield insights as you move from one stage to the next. Testing can help you figure out how to market your product or what features to develop next.

In short, the first concept test is just the starting point. There’s always more ideas to test and more to learn from your target audience. Keep involving your users along the way up to and even after launch.

Product concept tests are learning opportunities. Implement them with an open mind and an iterative approach. And if you want to learn more about how to create a concept testing survey, continue to the next chapter.