Three Split Testing Result.
Let's look at three tests and their results, each showing how we can not only improve results, but also learn more about what works, what doesn't... and what readers are looking for.
TEST 1: Testing the Impact of Current Events of Email Response Rates.
We set up a simple test with two emails, both of which were written to drive click-throughs to a site and convert leads into sales.
In email A we wrote the sales text within the context of an emotionally charged news story that was making headlines at the time.
In email B we wrote the email without specific mention of the event, but still alluded to "recent events in the news."
The essential difference between the two emails is that one mentioned the event by name, and the other didn't.
We tested our messaging to more than opt-in 337,466 email addresses.
We compiled the results after 12 days, although some clicks continue to trickle in.
Here are the results of the test after the first 12 days:
Email Copy Test Results
Email A Email B
Emails Sent 168,733 168,733
Clicks 5,119 4,395
Click-Through Rate (CTR) 3.03% 2.60%
Sales 175 122
Conversion (Click to Sale) 3.42% 2.78%
Conversion (Email to Sale) 0.104% 0.072
What You Need To UNDERSTAND: Email A (specifically mentioning the news story and events surrounding it) significantly outperformed Email B. CTR increased by 16.5% and overall conversion (email to sale) increased by 43.4%.
The email that disclosed the specifics of this news piece generated 53 more orders (an increase of 43.4%) than the email that only alluded to the events surrounding the story without mentioning specifics.
TEST 2: Testing a Specific Problem
In our second test, we believed that customers visiting our test site with an 800x600 or 1024x768 resolution monitor were not finding the relevant sales language for the primary site product unless they scrolled down that page.
We set up an A/B/C split to test this hypothesis:
Page A was the original page.
Page B featured slightly shortened data and used a "click here" anchor text to take visitors down the page. This page showed the order process on a 1024x768 resolution monitor and on an 800x600 resolution monitor it displayed the offer copy for the primary product.
Page C was a radical redesign in which the order process was partially viewable on 800x600 and higher. It used two columns to make more information available "above the fold."
Here are the results of our testing:
A/B/C Split Test
Page A Page B Page C
Percent of Traffic 34% 33% 33%
New Sales 244 282 114
Change N/A 15.57% - 53.28%
What You Need To UNDERSTAND: Page B outperformed the original page by 15.57%. Page C was a dismal failure.
Screen shots of the pages above are available below:
In a recent web clinic, we surveyed our audience as to which of these pages would perform the best. They overwhelmingly chose Page C, showing that what seems "intuitive" to most marketers is not always revealed as the best page after testing.
In this test, our hypothesis about important sales language appearing higher on the page proved correct. However, the two-column approach of Page C was ineffective.
TEST 3: Challenging Assumptions by Testing the "Obvious" and Learning from the Results
In this test we created two versions of a simple sales page online. Each page was approximately two screens in length and asked the reader to complete a short form in order to receive a free informational product.
In version A we added some personal elements to the page, including a photo of the writer, a personal introduction and a signature. In other words, we created a personal "sales letter" on the web page.
In version B the sales copy was largely the same, but without the personal elements: no photo, no salutation, and no signature.
Which version won? Conventional wisdom online suggested to us that the personal version would be the winner. Here are the results of our testing:
A/B Split Test
Version A (Personalized) Version B (Institutional)
Conversion Rate 34.6% 39.9%
What You Need To UNDERSTAND: Version B outperformed Version A by 15.3%.
In this case, our expectation that the personalized copy would perform better was simply not met.
There are two points to consider here:
First, if we had never tested these pages, we would have left money on the table by assuming the personal version would do better.
Second, this was only one test. What if we had taken the personal version and doubled the length of the copy as well, adding more of an individual sales pitch? What if we had changed the photo in some way, or repositioned it on the page?
KEY POINT: Each test provides answers. But each test should also stimulate further thought and additional rounds of testing to learn more.