The best usability testing questions

What types of questions do usability tests answer?

Unlike user testing, which is more open-ended and focused on users’ general attitudes and perceptions towards a product/feature, usability testing is concerned with answering targeted questions that get to the heart of user behaviors and website functionality. 

Usability tests are run for many different reasons:

• To validate a prototype

• To understand user behavior

• To uncover pain points, like bugs or functionality issues

• To get insights that help create a better overall user experience

Each situation requires slightly different feedback from your testers. Let’s look at some of the broader usability issues and corresponding questions you’d want answered to uncover valuable insights during a usability testing session.

How are users navigating your website? 

Website usability testing is imperative for understanding how users navigate your entire website and whether they have a good user experience. Consider looking for the following things:

• Are users following a clear path?

• How did they get from point A to point B?

• Are they finding the sections or pages they’re looking for?

How do users experience a particular page?

Another direction to take your usability testing is to examine the functionality of specific pages:

• Are users finding the information they need on the page? 

• Are they interacting with the features/elements you want them to?

• Does a newly launched element function the way it’s supposed to

What stops people from completing a task?

Usability testing can also identify barriers that keep users from completing a task and drive them away from your website:

• Are your user interfaces user-friendly?

• What, if anything, is stopping users from doing ____? 

• Do they need clarification on the wording or navigation?

20+ questions to ask in usability tests—and when to ask them

Usability testing has at least four stages: screening, pre-test, test, and post-test. We’ll cover how each stage provides an opportunity to collect important data from your participants, as well as how the specific goals of each stage dictate the types of usability testing questions you ask.

Before diving in, a quick reminder. Most questions can be asked using two different formats, depending on the type of data you want to collect:

• To collect quantitative data, use scales (e.g. from 0 to 10) and multiple-choice questions (e.g. “Pick between A, B, or C”)

• To collect qualitative data, ask open-ended questions (e.g. “What do you think of this pricing page?”)

When collecting quantitative data, following a yes-or-no answer with open-ended questions (sometimes called probes) is good practice. For example:

Question: Have you ever shopped online?

Follow-up questions: 

• If no → Why not? 

• If yes → How often do you shop online? and What products do you usually buy online?

Now that we’ve covered the basics of what questions to ask during usability testing, we’ll introduce a practical example. Let's run through the different testing stages and questions you would use if you needed to test your ecommerce website. 

Phase 1: screening

Before testing begins, you need to establish user criteria and decide what types of participants you want to include in your test. For example, you can select testers based on your user personas or target audience or focus on specific niches, such as people who do their online shopping on a mobile device.  

During the screening period, depending on which usability testing method you choose, you should have the opportunity to ask basic demographic and experience questions. In addition to using this data to evaluate potential test participants, you can use it later on in your analysis of the final results.

Examples of screening questions:

• How old are you?

• What is the highest level of education you've completed?

• What is your total household income?

• What is your profession?

• When was the last time you purchased an item online?

• Have you ever used [your website]?

Phase 2: pre-test

Once you've selected your participants, but before the actual usability testing begins, you have another opportunity to learn more about them, either with a questionnaire or a moderated interview.

During this stage, your goal is to learn about the knowledge and experience participants bring to the test—their backgrounds will inform their actions and opinions as they interact with your site.

The questions during this phase go beyond just collecting demographic data: they can be used to gather psychographic data about the participants' experiences, behaviors, and attitudes toward the product/feature being tested.

Expertise

These pre-test questions seek to evaluate the user's experience level with the actions they’ll be asked to perform:

• How often do you shop online? This will reveal a user’s buying habits and familiarity with online shopping.

• How confident are you with browsing, shopping, or other online tasks? This sheds light on their confidence level with online shopping.

• Which device(s) do you usually use for online shopping? This will help you determine which devices (e.g. mobile or desktop) users are more comfortable using. If possible, use their preferred device for testing to recreate a ‘natural’ environment. This also allows you to uncover specific usability issues on mobile or desktop versions of your site.  

Brand knowledge

These pre-test questions check the user's background and existing knowledge of your brand and products (or similar ones). Existing knowledge may sway their opinions or ability to use your product:

• Have you used this site before? This tells you about a user’s familiarity with your brand and knowledge of your site.

• Have you used a similar site before? This tells you about their familiarity with competitors.

• What would make you decide to buy ____ from [brand]? This gives you insight into how your competitors are succeeding and how you might improve your customer experience.

Phase 3: test

During the actual testing stage, your goal is to collect data that explains why users make certain choices while they complete tasks.

The test may be highly interactive, with users talking through their thoughts as they complete each step. Conversely, it could be silent, with users working independently and answering questions after each task or section. 

Either way, questions like the ones below should help illuminate what the user is doing, and why:

• I noticed you did ____. Can you tell me why? Follow up on any interesting behavior you observe during the test to better understand the thought process behind the user's actions.

• Was there any other way to ____? You’re trying to determine why the user did one thing instead of another.

• Which of these two approaches/options do you find best? Why? This is useful if you want to reveal the more appealing of multiple options.

• Can you tell me what you think of ____? By asking about specific aspects of the page (icons, menus, text), you can gather opinions on the UX design and functionality of your site so you can rework confusing components.

• How did you find the experience of using the website to complete this task? After every assigned task, ask this to learn how the user's experience and opinions evolve as they interact with your content. Here are some more focused examples:

  • What did you think of the layout of the content?

  • What did you think of the checkout experience?

  • What did you think of the on-page explanations?

Moderated vs. unmoderated, remote vs. in-person

You can implement different types of usability tests, depending on the budget and capacity your team has. First, you need to decide if you want to engage with participants in-person or remotely (keep in mind, in-person testing requires more logistics and resources than remote testing). 

You’ll also want to determine whether moderated or unmoderated testing sessions are appropriate (again, moderated sessions will require more prep and coordination than unmoderated sessions). Moderated user testing seeks to gain a deeper understanding of the whys behind your users’ behaviors. It involves a moderator who introduces and facilitates the test—they’re there to pose the prepared questions to users and answer any questions they might have.

On the other hand, unmoderated user testing hinges on observing and evaluating a very specific behavior or task. This method relies on passive online testing methods that allow you to observe how unsupervised users interact in a more natural environment than formal moderated sessions.

Communication techniques to employ during moderated usability testing

Whether you’re conducting in-person or remote moderated testing sessions, usability testing moderators need to be careful with how they interact with participants during the testing phase, so their words don't influence the test meaningfully. As a moderator, make sure you aren't asking any leading questions or asking questions in a way that could give the user hints or extra information about the specific tasks they need to complete.

This can be challenging when a participant has questions during the test. Here are some common communication techniques that moderators often use to respond to user questions during testing:

• The ‘echo’: repeating the user's last phrase as a question. For example, if the participant says, “This cart doesn’t work,” repeat it as “This cart doesn’t work?”

• The ‘boomerang’: using generic, neutral questions like, “What do you think?” in response to any of the participant’s queries

• The ‘Columbo’: channeling your inner Detective Columbo by trailing off on a sentence or question, so the participant can fill the silence by elaborating further

Phase 4: post-test

After a usability test, you have one final chance to ask the user questions that have yet to be covered elsewhere. This is the time to gather feedback about their impressions and opinions of your website and to get a feel for the overall user experience.

Common questions during this phase include:

• What was your overall impression of ____? Solicit feedback on the user's general opinion of your product. This is a broad question, so be prepared to dig deeper with some of the follow-up questions.

• What was the best/worst thing about ____? Get more specific feedback about the features that make your product/feature stand out.

• How would you change ____? This open-ended question is a good way to gather constructive feedback and ideas for future project iterations.

• How would you compare ____ to that of [a competitor]? This is an opportunity to see how different website details stack up against your competitors in the eyes of users.