• Marketing Land
  • Sections
    • CMO
    • Social
    • SEM
    • SEO
    • Analytics
    • Display
    • Retail
    • MarTech
    • Resources
    • More
    • Home
  • Marketing Land
  • CMO
  • Social
  • SEM
  • SEO
  • Analytics
  • Display
  • Retail
  • MarTech
  • Resources
  • More
  • SUBSCRIBE

Marketing Land

Marketing Land
  • CMO
  • Social
  • SEM
  • SEO
  • Analytics
  • Display
  • Retail
  • MarTech
  • Resources
  • More
  • Home
  • Newsletters
  • Home
Analytics & Conversion

5 Keys To Improving Your A/B Tests

If you want valuable customer insights, you need to test — and test some more. Columnist Jordan Elkind offers five approaches for more effective marketing experiments.

Jordan Elkind on June 8, 2015 at 11:59 am
  • More
AB-test-ss-1920-800x450

As a marketer, you’re often running tests to find which subject line gets the most email opens, what homepage content improves your search engine optimization (SEO), what button color leads to more purchases, and so on.

Sometimes however, you need to answer a different type of question:

“What can I do to cause an incremental improvement versus doing nothing at all?”

Let’s look at an example. Say you wanted to run an email campaign with a 20 percent discount offer to customers who are likely to want to buy knitwear. To determine whether or not this campaign is successful, you ultimately need to know how many people made a purchase from this campaign that wouldn’t have done so if they never got the discount email.

If 80 people buy a new knit top, but almost all of those people would have done so anyway (except now they got a nice 20 percent off coupon), then you’ve actually hurt your company by giving discounts where they weren’t needed.

However, by properly designing and executing an experiment with a control group (or holdout sample) you are able to predict with great accuracy whether or not the campaign was effective. In short, you need to take a random subset of the population and do nothing to them, and then look at what they do compared to the group that received the treatment (an email campaign in this example).

While it can be hard to prioritize experiments, they can give you a better idea of the true return on investment (ROI) for your marketing, which can pay off big in the long run.

The following are five keys to effective marketing experiments.

1. Keep The Control And Test Groups Balanced

Stay disciplined about control groups — the temptation is there to shrink the control group down to maximize the earning potential of the test. The problem, however, is that without an adequately sized control group, deriving meaningful results and conclusions can become next to impossible since you don’t have a good read on baseline customer behavior.

Start out with a 50 percent control group, then cut it down as you begin to have a better sense of what works and what doesn’t.

2. Make Sure The Control And Test Groups Look The Same

Your control group needs to be the same as the test group. Not “kind of” — it needs to be identical. If you’re trying to see how a 10 percent discount works for mothers in Texas, then both your control and experimental groups need to be mothers from Texas.

To do this, you need to define your segment, and then randomly assign members to the test and control groups. Some email service providers can do this for you, otherwise you’ll need to use Excel or another tool to create the random groups yourself and then upload accordingly.

3. Eliminate Other Variables

Since you’re trying to see the impact of one idea, and only one idea, then you need to make sure that the test and control groups are otherwise identical in treatment. If one group gets a daily email about socks, then the other group needs to get that same sock-loving email; otherwise the results are tainted and therefore meaningless.

4. Keep It Simultaneous

Things change — strategies shift, seasons change, competitors rise and fall. Given these facts of business, you need to make sure that your test and control groups are being tracked at the same time. You can’t compare last month’s holdout group to this month’s test group.

5. Make Sure Results Are Trackable

The whole idea is to be able to measure the test and control groups against each other, so make sure you know what you’re tracking and how you’re going to track it.

And since the control group isn’t getting any emails, you can’t use clicks or opens to compare. (Customers can’t open an email they never received.) Look at revenue, orders, site visits, conversion rates, and so on during the test period to see whether the campaign did well enough to merit further investment.

To Recap

To measure the real impact of your next marketing campaign (and see its true incremental effect), make sure to have a holdout group that doesn’t receive the campaign so that you can properly compare results. Happy testing.


Opinions expressed in this article are those of the guest author and not necessarily Marketing Land. Staff authors are listed here.



About The Author

Jordan Elkind
Jordan Elkind heads the product team at Custora, an advanced customer analytics platform for e-commerce retailers. Prior to joining Custora, he earned an MBA from Wharton and worked in marketing analytics at Citi Cards.

Related Topics

AnalyticsAnalytics & Marketing ColumnChannel: Analytics & ConversionEmail MarketingHow To Guides: Email MarketingStatistics: Email Marketing

We're listening.

Have something to say about this article? Share it with us on Facebook, Twitter or our LinkedIn Group.

Get the daily newsletter digital marketers rely on.
See terms.

ATTEND OUR EVENTS

Next Event: Sept. 14-15, 2021

Available On-Demand: March 2021

Available On-Demand: October 2020

×

Attend MarTech - Click Here


Learn More About Our MarTech Events

April 13, 2021: SMX Create

May 18-19, 2021: SMX London

June 8-9, 2021: SMX Paris

June 15-16, 2021: SMX Advanced

June 21-22, 2021: SMX Advanced Europe

August 17, 2021: SMX Convert

November 9-10, 2021: SMX Next

December 14, 2021: SMX Code

Available On-Demand: SMX

Available On-Demand: SMX Report

×


Learn More About Our SMX Events

White Papers

  • Nielsen Annual Marketing Report: Era of Adaptation
  • Global Social Media Advertising Trends Among Retailers in 2021
  • 2020 Hispanic Market Report
  • Delivering Audio Success: Industry Trends in Streaming Audio & Podcast Performance
  • Ten Tips for Acquisition Email Success
See More Whitepapers

Webinars

  • Drive Customer Engagement with the Power of Personalization
  • 7 Use Cases That Prove Why You Should Implement DAM
  • Accelerate Your SEO & Content Marketing Program with 4 Key Milestones
See More Webinars

Research Reports

  • Local Marketing Solutions for Multi-Location Businesses
  • Enterprise Digital Asset Management Platforms
  • Identity Resolution Platforms
  • Customer Data Platforms
  • B2B Marketing Automation Platforms
  • Call Analytics Platforms
See More Research

Attend SMX For Only $149

h
Receive daily marketing news & analysis.

Channels

  • MarTech
  • CMO
  • Social
  • SEM
  • SEO
  • Mobile
  • Analytics
  • Retail
  • Display

Our Events

  • MarTech
  • SMX

Resources

  • White Papers
  • Research
  • Webinars

About

  • About Us
  • Contact
  • Privacy
  • Marketing Opportunities
  • Staff

Follow Us

  • Facebook
  • Twitter
  • LinkedIn
  • Newsletters
  • RSS
  • Youtube

© 2021 Third Door Media, Inc. All rights reserved.