What is CRO Testing? Benefits, Types & Examples

This post shows how to do the most essential conversion optimization (CRO) testing, and how it can help you find the strengths and weaknesses of your website.

Also, you will see examples of how other successful online businesses do CRO tests and optimize conversion rates up to 25% for one campaign.

Get started:

• What is a CRO test?

• Types of CRO tests

• Benefits of doing CRO tests

• How to perform a CRO test

• Examples of CRO tests

• Tools for CRO testing

If you're interested in CRO, consider these guides for later:

• A beginner's guide to CRO: What is CRO? tips for beginners

• See how businesses use CRO: Examples of CRO in action

• Tips for online stores: Conversion optimization for ecommerce websites

What is a CRO test?

A CRO test is an experiment on a website that involves modifying its elements to increase the number of users that complete the desired conversion action, such as buying, signing up for a newsletter, or creating an account.

CRO tests typically involve comparing two versions of a web page or campaign using specific qualitative and quantitative metrics, including conversion rates (CVR), click-through rates (CTR), bounce rates, number of orders, or collected leads.

Types of CRO tests

CRO tests range from analyzing on-page copy to something as complex as creating personalized experiences for different segments of website visitors.

A/B testing

Minimum traffic requirements: about 1,000 unique visits per variant

A/B testing involves comparing two different versions of a web page, a page element such as a button, or an onsite campaign such as a popup, to see which one produces more conversions. This type of CRO testing often evaluates the number of orders, average order sizes, captured leads, or clicks.

In most A/B tests, one variant (usually called variant A) is shown to 50% of the audience while the rest see the changed version (variant B). Thanks to this simplicity, A/B testing can often be easily done without experience.

Example: analyzing two web page versions with different hero images and positions of the CTA buttons

A/B/n testing

Minimum traffic requirements: about 1,000 unique visits per variant

A/B/n takes A/B testing one step further by adding more than two variations. This type of CRO test is mostly done by online businesses with a significant amount of traffic to split up into more than two audiences.

The biggest advantage of A/B/n testing is the ability to evaluate more than one idea in one run. That's why the opportunity to generate more valuable insights can be much higher compared to A/B.

Example: testing three different versions of a product page to see which one generates more orders

A/B testing vs control

Minimum traffic requirements: about 1,000 unique visits per variant

This CRO test involves testing multiple campaigns against control groups while getting useful data on conversions for each. For example, you can a) test the impact of two or more different promotions against a baseline or b) find out if a campaign will improve conversions on a page or not. Also, the test produces data such as attributed revenue per visitor, pageviews per visitor, and time they spent on the website.

Example: testing two different promos and comparing them to the baseline

Multivariate testing (MVT)

Minimum traffic requirements: about 10,000 unique visits per variant

Multivariate testing is often used when A/B/n test results leave room for improvement. It involves changing multiple elements on a webpage, and analyzing how different combinations of elements affect conversion rates.

This type of CRO test is more complex, requiring advanced statistical analysis to draw accurate conclusions. However, it can provide valuable insights into the most effective combination of design elements for a webpage.

Example: testing different variations of headlines, images, layouts, and calls-to-action on a landing page

Heatmap and visitor session replay testing

Minimum traffic requirements: about 1,000 unique visits per variant

Heatmaps and session replay testing allows you to visually analyze user behavior and interactions on a webpage. Online businesses use this type of CRO testing to see points of confusion, the most popular areas, and the time spent on pages.

Besides qualitative data, heatmaps and visitor sessions also help with getting such quantitative insights like clicks in buttons, error encounters, click-through rates, average scroll depth percentage, and time before the first click.

Customer feedback surveys

Minimum traffic requirements: none

Customer feedback surveys are conducted onsite (via forms and popups) or offsite (emails, interviews) to collect insights into what users seek on your site by asking specific questions. Although website surveys are usually used in the research stage of CRO projects, but they can also be used for testing purposes.

For ecommerce stores, for example, this type of CRO test provides valuable information about customer pain points, preferences, and expectations. The answers can help with finding areas for improvement and creating more types of CRO tests to address those issues.

Example: Surveying customers who abandoned their carts to understand why they left without completing the purchase

Multi-armed bandit testing

Minimum traffic requirements: 4,000 unique visits per variant

Multi-armed bandit testing optimizes the use of traffic by dynamically allocating more visitors to better-performing variations. Say, if you’re testing different product page layouts, the testing tool quickly prioritizes the one driving more sales. The prioritization happens within hours to a few days, depending on traffic volume.

With this type of CRO test, you waste less traffic on low-performing options, optimizing results faster. The more visits and conversions you have, the faster the experiment adapts, continually adjusting to maximize performance in real time.

Benefits of CRO testing

Essentially, CRO testing helps your marketing ideas turn your website into a persuasion machine that meets the needs of visitors.

Get more orders

By optimizing the pathways to purchase and identifying which site elements boost engagement, you can make data-driven decisions that contribute to higher ecommerce sales. This helps you understand your visitors' behaviors better, enabling you to tailor the user experience to encourage more purchases.

Increase average order value

With a good CRO testing plan, you can find ways to increase the average amount spent by each customer. Those conversion tactics involve testing different pricing strategies, upselling techniques, onsite channels, or bundling products together.

Reduce customer acquisition costs

CRO testing can lower customer acquisition costs by improving your website's efficiency in converting visitors into customers. When your site's conversion funnel is well-tuned to guide users towards making a purchase or converting them in other ways, you spend less on attracting new visitors.

Squeeze more leads and opt-ins from your visitors

Testing different incentives, designs, campaign display timing, page placements, numbers of signup fields, and other factors can help you improve lead generation. That could be as easy as launching an A/B test of two popup forms with different designs and discounts.

Create personalized browsing and shopping experiences

CRO testing can show you how to make personalized experiences tailored specifically to the interests and behaviors of individual visitor groups. This enables the creation of customized product recommendations, offers, or content, which not only enhances the browsing experience but also builds customer loyalty.

Steps to perform a CRO test

Performing a CRO test involves a series of structured steps, each crucial to ensuring that your test results are both significant and actionable.

Step 1: Define test goal(s)

The goal here is simple: what would you like to achieve?

More orders, email signups, clicks, or form submissions?

Choose your goal and make it measurable. For example, the goal might be "to increase newsletter signups with popups and forms by 10%"

However, hold off on starting CRO testing for now—

Do some research first.

If we continue with the newsletter signups example, you might want to check the designs and incentives used in previous newsletter campaigns. Was there something that worked well? How many emails they collected in total? Did they appear on the page too quickly?

Then, go to your competitors' sites and check out their campaigns for inspiration.

Finally, compile all your findings in one place and brainstorm to develop a hypothesis.

Step 2: Formulate a hypothesis statement

Come up with a hypothesis based on your collected data. This should outline what changes you're making to the current web page designs or campaigns, the expected performance improvements, and the rationale behind those changes.

The hypothesis could be as simple as one sentence:

"If we switch our newsletter campaign to an exit intent popup with a discount and countdown timer (the change), we should boost signups by 10% (the expected impact) due to the urgency of the timer and the discount incentive (the rationale)."

Step 3: Create design variants for testing

It's time to create your test variants. For example, if your hypothesis involves changing the design of a newsletter signup form, create multiple versions with varied elements like product images, headlines, or incentives.

Use design or marketing tools that allow you to quickly create, duplicate, and modify these variants. The designs should be prepared for testing in a CRO tool or created directly within the tool.

Step 4: Run the test

Before running the test, set up the monitoring, such as goal, event, or revenue tracking. Once that's done, deploy and run your CRO test until the minimum traffic requirements are met.

Be ready to pause the test, too, as some potential negative impacts on website speed or user experience may happen.

Step 5: Analyze the results

Once the test hits the minimum traffic requirements or one variant significantly outperforms the other, review the performance indicators. If you see a clear winner, then implement it right away on your website.

If the results aren't statistically significant (eg the test didn't meet minimum traffic requirements), consider extending the test duration or coming up with a different hypothesis.

Also, in case the difference between the variants is small, you may test different designs or implement the better one, even if it means getting only a slight increase in conversions.

Step 6: Consider more tests

Conversion optimization is an ongoing process, so you should look into more opportunities to increase your website conversions.

Using the newsletter signup as an example, consider adding an extra step to collect phone numbers or an extra field to capture customer names.

Examples of CRO testing in action

Here are a couple of mini case studies showing how CRO tests can make a difference.

OddBalls generated 4.5x more emails with onsite notifications than social media posts

Goal: build a waitlist for a new product

“Our challenge was to generate sign ups for a completely new product type of ours that we had not previously launched before. This was a new way for us to engage in creating sign ups for the business.”

Dan Mitchell, Ecommerce Manager OddBalls

OddBalls primarily offers underwear, but they consistently expand their product line. Not so long ago, they introduced fantastic indoor hoodies, promoting them through a few marketing channels. Facebook posts and onsite notifications were among those channels.

In this CRO test, OddBalls made two marketing campaigns: an onsite notification and a Facebook post. The results showed that the post collected 150 emails while the onsite notification captured 693 emails:

Besides collecting emails, the onsite notification was also seen by over 5,000 visitors, helping to raise even more awareness (below). These figures converted to an impressive 25% CTR and a 3% CVR.

More: read OddBalls case study

Asphalte achieved a 25% CTR of product survey campaigns

Goals: increase lead generation through product surveys; introduce visitors to the brand's mission and values

"We realized that customer lifestyle value was higher after a customer has learned about our mission and how we work."

Ben Mateo, Head of Growth, Asphalte

Asphalte makes clothing based on answers their customers and visitors give in product surveys, which introduce the brand's mission: zero overproduction. To drive as much traffic to the survey as possible, they tested popup forms with different visuals and sizes.

Here's one variant of the campaign (the smaller size, a different visual):

And—

Here's the second variant (a full-screen campaign, a different headline and visual):

Faguo also experimented with popup timing and targeting. Specifically, displaying campaigns was delayed by 30 seconds while many other web pages were excluded from this CRO test.

The A/B tests with different versions paid off nicely, as Faguo increased the CTR from 15% to 25%, which also helped to collect more 4,000 leads in one month.

Read full CRO case study (also available in French)

CRO testing tools

CRO tools enable and automate monitoring, evaluating, and improving website conversion rates with analytics, A/B testing, experiments, heat maps, visitor recordings, or other features.

• Wisepops. An affordable CRO platform for monitoring lead capture and other marketing personalization campaigns with beginner-friendly CRO experiments and A/B tests. Includes lead capture forms, AI product recommendations, and ecommerce-focused features like built-in Shopify campaign targeting.

• Smartlook offers advanced website heatmaps and user analytics. Its seamless integration with popular ecommerce platforms and comprehensive visitor recordings make it an excellent choice for both beginners and seasoned marketers.

• Google Analytics (GA) delivers crucial data-driven insights through a variety of report types, such as real-time, audience, acquisition, behavior, and conversion reports. While Google Analytics shows what happens on a website, it should be used with other conversion rate optimization tools to understand why users behave as they do.

Summary

CRO testing offers a powerful approach for enhancing website effectiveness and boosting conversion rates. By systematically experimenting with different variables, you can gain valuable insights into user behavior and preferences, leading to more informed decision-making.

Learn more about increasing conversions:

• Best ecommerce marketing strategies

• 20 ideas to improve conversion rate on Shopify

• Lead generation ideas and strategies for B2B businesses