AB Testing Research: Do Your Conversion Homework

IF we had to pick one thing that has made Conversion Sciences a successful AB testing agency, it would be this: We are very good at picking what to test.

This isn’t because the team is made up of geniuses (except you Brian, we all know you’re a genius). It’s because we have a consistent methodology for conducting AB testing research. In other words, we do our homework.

Like we talked about in our our rundown of the best AB testing tools, “your AB tests are only as good as the hypotheses you are testing.”

With the proper research, we can consistently make better hypotheses, leading to more profitable testing results and a better experience for our visitors.

100 Million Neurons in Our Guts

Would you believe there are 100 million neurons in the human gut? This concentration is second only to our brains, even prompting scientists to refer to it as our “second brain”.

While we don’t use our gut to make conscious decisions, it can greatly influence our mental state and is likely the reason we have “gut reactions” or “gut feelings”.

There are times when “going with your gut” makes sense. That may happen when you don’t have any other options, or when your gut is trying to tell you something but you are unable to rationally identify it. If there is no information available to you, your gut may be a good second opinion for your brain.

On the web, there is rarely a need to go with your gut due to lack of information. So. let’s redefine these terms.

• Whenever someone says “My gut reaction is…” you should hear, “I don’t really know. Let’s do some more research.”
• Whenever someone says, “I have a gut feeling that…” you should hear, “I don’t have enough information. How can we better inform ourselves before making this decision?”

We are living in a golden age of digital marketing information. With such easy access to research methods, there is no good reason to ever go from the gut on web design, copywriting, value proposition, or conversion optimization.

You don’t need your intestines to design your website.

After all, the primary output of a healthy gut is… well… crap.

What research skills can keep you from resorting to your colon for inspiration?

To answer that question, we worked with KlientBoost to capture many of the key AB testing research methods and enjoy the satisfying feeling of winning AB tests.

Research + Framework = Growth

AB testing research feeds an AB testing framework for test results that are consistently positive and repeatable. Feed it well, and it will poop out revenue growth month after month. This is the only resemblance to your gut I could think of.

It doesn’t make sense to test an idea without some evidence that it will make a difference. Good research is full of nutrients, vitamins and fiber. And that is the last time I’ll refer to the digestive system in this article.

Related: What Keeps Visitors from Converting on your Site?

The Heart of AB Testing Research

Before we get into the details, it’s important to understand the core of testing research, and ultimately, the core of conversion optimization itself:

“The definition of optimization boils down to understanding your visitors.” – Brian Massey

Optimization is just a fancy word for bettering our understanding of our customers and giving them more of what they want.

Behavioral data is the best, most reliable source for split testing. With it, we can eliminate tripping points and optimize the visitor’s experience.

We find this data in our analytics databases. But you may notice that much of our AB testing research will not be behavioral. And that is fine.

Generally speaking, there are two types of research:

1. Quantitative Research
2. Qualitative Research

Both kinds of research will provide us with good data to form hypotheses and test variations. Let’s review their pros and cons.

Quantitative Research Data for AB Testing

Quantitative data is generated from large sample sizes. Quantitative data tells us how large numbers of visitors and potential visitors behave. It’s generated from analytics databases (like Google Analytics), trials, and AB tests.

The primary goal of evaluating quantitative data is to find where the weak points are in our funnel. The data gives us objective specifics to research further.

There are a few different types of quantitative data we’ll want to collect and review:

• Backend analytics
• Transactional data
• User intelligence

Understanding Qualitative Research

Qualitative data is generated from individuals or small groups. It is collected through heuristic analysis, surveys, focus groups, phone or chat transcripts, and customer reviews.

It can uncover the feelings and reactions of your user experience as they visit a landing page and the motives or reasons why they interact with your website in a certain way.

As qualitative data is often self-reported, we should analyze it with grain of salt. Humans are good at making up rationalizations for how they behave in a situation. These qualitative research studies are not conducted at great scale, therefore reducing their statistical significance. However, it is a great source of test hypotheses for future testing that can’t be discerned from quantitative behavioral data.

There are a number of tools we can use to obtain this information:

• Surveys and other direct feedback
• Customer service transcripts
• Interviews with sales and customer service reps
• Session Recording
• Heat maps

In summary, quantitative data tells us what is happening in our funnel and qualitative data tells us why visitors behave the way they do. Both types of data give us a better understanding of what we should test.

Usability and User Experience

Why are we going through all this data to perform ab testing research? Because two of our key goals are to evaluate our website’s usability and user experience.

• Usability deals with how easy it is for someone to learn and use the functions of our site. If we can make any part of that customer journey easier or more intuitive, we are increasing Usability.
• User Experience deals with the emotions and attitudes users experience as they use our site. If we can make the customer journey more enjoyable, we are improving UX or User Experience.

While these two concepts often go hand in hand, they are not the same, and both need to be kept in mind when collecting data.

The Importance of Segmentation

It’s not enough to simply know that “visitors” are doing ____ when they visit a given webpage or flow through a given funnel.
Which visitors?

• Are they on mobile or desktop?
• Are they here via paid ads or organic search?
• Are they using Chrome or Firefox?
• Did they click-through via a Facebook post or a Tweet?
• Are they a new or returning visitor?

In order to properly understand and evaluate our visitors and customers, divide them into strategic segments to understand the differences across each segment.

It’s especially important to know what these key segments are before we run our AB tests, because otherwise, our tests won’t tell us anything about them.

Follow the Proven System A/B Testing Agencies Use

AB testing research is a fundamental part of any proven CRO framework, and it’s an important part of what separates an ROI-generating A/B testing agency from a waste of money.

As you finish up the year and move into a brand new one, it’s time to take things up a notch. In the past, hundreds of businesses drastically improved their bottom lines via a proven, systematic CRO process.

Why not join the party? Take our free gift and click here to schedule a call with one of our CRO professionals.

References

Think Twice: How the Gut’s “Second Brain” Influences Mood and Well-Being