Product experimentation: what is it and why does it matter?

Product experimentation lets you measure the impact of the products you’re building—testing small changes at scale, occasionally failing, and using the results to learn, iterate, and optimize. 

It can be tricky to master, especially for those who don't know how to dose their efforts, manage risks (perceived or real), or have difficulty getting team members into this mindset. But when done right, experimentation adds heaps of value to product-building and goal-setting processes.

This guide will introduce you to product experimentation. In later chapters, we’ll show you how to conduct different experiments with a step-by-step framework, discuss how product validation helps with experimentation, and outline the full tech stack of product experimentation tools teams should use to conduct useful tests.

What is product experimentation?

Product experimentation is the process of continually testing hypotheses for ways to improve your product. It’s a scientific approach to product management, geared towards understanding how customers use your products and identifying ways to improve them.

Great products are built on experimentation. It helps product teams spot specific ways to improve the user experience (UX) and deliver small incremental changes that have a big impact. 

What types of experiments can take place in product development?

By running regular experiments, you’re constantly exploring possibilities for new features and product improvements. Generating that feedback loop with your users allows you to get deeper into your customer’s experience, so you're consistently delivering the best possible solution—and a delightful one, at that.

The most common forms of product experiments throughout the product development cycle are:

• A/B testing: this UX research method implies running two different versions of a product, website, app, or feature to determine which performs or converts the best. A/B test measurable elements of your website that can affect a visitor’s decision to convert—including page structure, registration and sign-up forms, and calls to action (CTAs).

• Multivariate testing: similar to A/B testing, but with multiple variables changing at the same time. This methodology is most useful when you have alternatives with multiple components, and you're trying to find a combination of variables to drive the results you’re looking for.

• Funnel testing: another variation of A/B testing, this method involves making changes across multiple pages to see how users flow through various stages of the customer journey. This is most useful when you have components that need to stay consistent across pages (like navigation) or are testing the user journey (like introducing new paths or shortcuts).

• Split testing: similar to A/B testing, except instead of having the two web pages or website variations competing against each other, the traffic is equally split between the existing variations. This methodology is mainly used in instances where changing all assets at once can be costly, and there are potential SEO penalties for having multiple versions of the page.

The components of a great experiment

Making decisions is a critical part of a product manager's job. The scientific method of running regular product experiments ensures that you make data-informed decisions about how your product—and the customer experience—should evolve. 

So that’s why you need to be running experiments. But what makes a good experiment?

The good (and bad) news is that you can experiment with pretty much anything: new features, changes to UX, even the copy that conveys what your customer should do next.  

However, if you want your experiments to generate valuable insights, they need to include these specific components:

The problem 

A good experiment solves a real user problem. But deciding what to work on shouldn’t be a shot in the dark. Use data analysis, market research, and customer and product experience (PX) insights to identify user problems and prioritize your roadmap with unbiased, reliable user data.

The (possible) solution

What features or changes can you offer to solve the problem for your users? Valuable experiments provide one or more solutions that you believe will create value for users. Be open-minded—you know there’s a chance your experiments will flop, but you’ll still learn, even if they do.

The benefit

What is the outcome you’d like to see for your users? Or what business goal are you trying to change? Experiments need an upfront definition of success. Good experiments know the benefit they’re giving to users, understand what customers want and care about, and see how results point to flaws with your current product strategy.

The users

Who are you trying to help? How big is that audience segement? And how do they typically behave on or interact with your site? Good experiments begin with a clear understanding of users' current behavior, and a definition of the cohort who will experience the test. 

The data

How will you measure success? Good experiments are built around an ultimate goal or metric you want to improve. Meaningful, actionable customer feedback about the impact of your product changes is the ‘secret sauce’ of a valuable product experiment.

To successfully innovate, companies need to make experimentation an integral part of everyday life. Once you understand the five components above, you can start working on the three key layers of great experiments:

3 key layers of effective product experiments

1. Setting realistic experimentation goals

The main goal behind any experiment is to improve your product for your users. As results are reported, teams can use the data to make decisions, understand key learnings, and adjust course if needed.

The bottom line is, to embrace a product experimentation culture, you have to look at every failure as a learning opportunity. This encourages your team to take risks, be curious, and experiment as a day-to-day process.

Remember, knowing what not to do also makes your product better. Create a hypothesis and test it. If it’s not valid, learn from it and move on to the next one.

2. Getting buy-in and aligning with your team 

The most innovative companies have embedded experimentation into their culture and make data-backed decisions to implement and create new features, improvements, or products. 

As a product manager, the best thing you can do is to empower your whole team (including senior stakeholders) to think and act creatively, and make sure everyone understands that experimentation is a worthy investment. 

Creating a shared understanding amongst the team on how users experience your product will help you prioritize brilliantly.

3. Establishing processes, infrastructure, and technology

Part of a product manager’s job is to establish essential processes and technology that make experimentation easy. Think through what tools you use to influence user behaviors: copy, design, UX, or features. Each solution is a test worth running.

Establish a process that defines how you run an experiment, as well as how you analyze and implement its results. By training everyone on how experiments work and sharing the same tools, you build trust in the process and the results.

Is product experimentation the same as product validation?

In short, no. Instead, Senior Product Manager Andrei Beno sees product experimentation as a form of validation.

The main difference to keep in mind? Andrei says, “Validation is most useful to assess a new feature or product, while experimentation is most useful to assess iterations and improvements to a feature or product.”

Validation is often most helpful at the earliest stages of a product’s development, when teams need to gather information on whether the product actually provides enough user value and makes sense to build. 

Experimentation, however, enables product teams to improve upon the value that is already developed, to make the customer experience more delightful. Your product team can leverage running regular experiments wherever they are in the product development cycle: they can test an idea before launch, or they can improve a feature after it’s rolled out to your customers.

Note: we have a full guide on product validation coming your way soon, highlighting how the process of product validation can tie into product experiments and why the two areas are closely aligned.

What are the benefits of product experimentation?

Experimentation sits at the heart of product development. 

Product experiments let you validate your ideas and initiatives with real, unbiased insight about what your users need.

A structured approach to incremental product experimentation helps your team:

• Identify what users love, hate, and feel indifferent about so you can fix issues fast and spot product opportunities that’ll really move the needle. 

• Save money and resources by understanding the real impact of your changes and getting a quick response, with minimal investment and disruption.

• Reduce the risks of future releases by seeing the connection between the trends in your user cohorts and the real human behavior driving those trends, validate your hypotheses, reduce assumptions, and increase confidence in your ideas.

• Get wider stakeholder communication by convincing stakeholders to prioritize the experience over the aesthetics and getting buy-in to prioritize product opportunities that you know will create emotionally engaged users.

Ultimately, good experimentation enables companies to become more aligned in their delivery and more connected to their mission, values, and goals.

With the right combination of a growth mindset, data-driven decisions, and useful tools, you can drive product growth and make your users successful. Companies that embrace product experimentation are able to capture highly accurate data about their customers and use it to:

1. Quickly identify problem areas and opportunities for optimization

By evaluating ideas and making decisions based on data, product managers and their teams can eliminate the guesswork and avoid wasted time and effort.

Thinking about product design and development with an experimentation mindset is an asset for any product team. It can help product teams move faster and with more confidence at all stages of a feature’s life cycle, and be able to iterate much more quickly.

One easy way to check for experiment opportunities is to use an on-site survey to find out what users need and are expecting from your products. Using JavaScript triggers, you can implement a survey pop-up for users who click a certain element or scroll to a set point.

Hotjar lets you gather responses quickly, which you can then categorize into general themes so you can quantify the most common issues and requests.

Analyzing responses from open-ended questions makes it possible to convert qualitative data into quantitative insight and use it to prioritize the product roadmap. Hotjar Highlights helps product teams group insights to organize and curate data to make confident decisions and prioritize brilliantly to reach business and product goals.

This sort of experiment helps you learn more about your user base and apply those learnings to other parts of the product. Most importantly, it helps you craft the future of your product.

2. Spot friction points and use insights to bring hypotheses to life

Good experiments help you build products using the strongest form of evidence—not opinion. Experimentation involves low investments, both financially and emotionally, and lets you test more high-risk ideas, in a low-risk way. 

Because experiments are, by nature, measurable and reversible, they’re a huge opportunity to test ideas that are bolder than you’d dared to test before.

For example, maybe you think your idea to create a new feature from scratch is great, but you're worried it’s something that your users may not use. Gathering data from experiments will help guide the direction of the product and feature iterations. 

Running an A/B test with a product experience insights tool like Hotjar (hi there 👋 ) lets you record interactions between individual users and the product, so you can see how it’s working, where people are getting stuck, and collect feedback to learn what they’re thinking about the change.

If the result is successful, it's a clear sign that you should put more effort into implementing your idea across a wider level. If not, experiments help you see the light at the end of the tunnel. 

After all, it’s easier to turn off an experiment than it is to write off an entire feature launch. 

3. Test solutions and prioritize based on user needs

A good experimentation process is a never-ending cycle. Every result—both positive and negative—gives you insights to create a new hypothesis. Once an experiment is complete, the results need to be understood, communicated, and implemented.

Communicate the results with the rest of your team and create an action plan for what you’ll do with them. Based on that, you can make informed, data-backed decisions and better prioritize your roadmap so it resonates with users and aligns with business goals.

Oh and don’t forget to celebrate the small wins! Showing progress keeps everyone excited and motivated. Using a tool like Hotjar gives you a central place to keep track of your experiment results and share the process with the rest of your team.

When to run product experiments—and when not to

Experimentation is a feedback loop with your customers. It moves beyond conversions or customer acquisition, to improving adoption and retention so your product becomes indispensable in your customers’ lives. 

However, not everything can (or should) be tested rigorously before making a decision. Here are a few instances when a product team should run experiments:

• When you need to understand how your product can do an even better job of satisfying your customers' needs 

• To understand how your customers' needs have changed since you initially launched your product

• When you need to validate ideas for future projects with reliable data, based on unbiased experiences

• To troubleshoot issues with reliable user insights and make the right prioritization calls

…and times when experiments won’t truly be helpful:

• To establish product-market fit or assess if the product provides enough user value and makes sense to build. Instead, use product validation to highlight the core factors you need to weigh up when making these decisions.

• If you’re just looking for successful tests with vanity metrics. Overemphasizing the importance of successful experiments may cause employees to only focus on solutions they already know will work and avoid testing ideas that they fear might fail.

• If opinions are valued more than data. Nothing stalls innovation faster than a so-called HiPPO or ‘highest-paid person’s opinion’. The empirical results of product experiments must prevail when they clash with strong opinions, no matter whose opinions they are. Getting executives in the top ranks to abide by this rule isn’t easy, but it’s vital that they do.

• If you’re looking to make strategic management decisions. Some things are almost impossible to conduct tests on—for example, strategic calls on whether to acquire a company.

Use strategy, experience, and leadership to drive your team forward through the moments where experimentation can’t be used. In time, you’ll start knowing when to use each tool and when to trust your gut over the available data.

Hotjar helps make experiments more successful

Product experiments should be easy to review and understand. They are a strong foundation for maximizing the learning opportunity—especially for those who have doubts about whether a particular idea is a good choice for customers.

Hotjar helps you analyze product experiments and makes feedback easy to collect, process, and share. Get inspired with user-centric insights, discover opportunities, reduce assumptions, increase confidence, prioritize fixes, get buy-in from stakeholders and align with your team—all while using just one product.

The product experience insights you collect using Hotjar tools are an intuitive, visual way to understand the true experience users have in your product; unbiased, in-context, and at scale:

• Heatmaps help you collect data on how people use pages of your site after a change and can be paired with A/B testing.

• Session Recordings let you dig deeper and see how users behave on a page, and what happens when they get stuck or before they exit.

• Surveys and Feedback widgets are useful when you want to know why users behave a certain way on your site, or you want some insight into their thoughts or feelings when they come across your experiment.