OUR BLOG

A/B Testing: Seeing the Forest

Written by Amy Gyoba on January 16, 2012

Have you ever heard the saying “not seeing the forest for the trees”? It means that someone doesn’t see the forest (the big picture) because they’re too busy looking at the trees (the details). We experienced this problem while performing A/B testing and want to share our lessons learned. In this case, it took some time to figure out why we weren’t getting conclusive results—we had been focusing too much on the small details.

Defining A/B Testing

A/B or split testing is exactly what it sounds like: testing one version of  an ad or webpage against another one to see how a change to one component affects the conversion rate. Conversion, the desired user action, is defined before testing. It’s a key part of split testing because it determines which version is more successful.

Performing the Test

Last year we performed A/B testing on a client’s homepage to check the effectiveness of the booking call-to-action. On the original homepage, the button was placed in a static image at the bottom-right corner. In our variation, we placed the button in a prominent sliding banner.

To determine which one drew a stronger response, we needed to define conversion for this test. We decided that it occurred when a visitor booked the client’s services. The homepage that elicited the highest number of bookings would be the most successful.

Solving the Problem

As the testing went on, we monitored the numbers, but there weren’t any significant results. We brainstormed possible explanations and tweaked aspects like the button label and placement in our version. Even with these modifications, the results continued to be inconclusive.

Then a team member found the solution; rather than adjusting the design of the page, we should have examined the test plan. Quite simply, we had come up with the wrong definition for conversion.

Remember, we had originally defined it to be when a visitor booked the service. However, between clicking the call-to-action and submitting the booking request, there were several factors—beyond our control—that might have influenced someone’s decision to book. They may have made the decision based on the wording of the booking page or the workflow to book the service; this decision was not necessarily dependent on the call-to-action on the homepage.

The actual definition in this case was much simpler: conversion meant that the visitor clicked the call-to-action. And once we changed the measurement of success, we began to see clearer results for the split testing.

Seeing the Forest

When facing a problem, it’s so easy to focus on fixing the small details that we don’t consider the big picture. For our split testing, we thought the problem was in our homepage design; we didn’t consider our plan to test it. A key part of split testing is comparing conversion rates; if that definition is inaccurate, the results will be too. Once we were reminded to step back and look at the big picture, we found the solution to our problem.

Image Credit: zakwitnij/cc