• Marketing Land
  • Sections
    • CMO
    • Social
    • SEM
    • SEO
    • Analytics
    • Display
    • Retail
    • MarTech
    • Resources
    • More
    • Home
  • Follow Us
    • Follow
  • Marketing Land
  • CMO
  • Social
  • SEM
  • SEO
  • Analytics
  • Display
  • Retail
  • MarTech
  • Resources
  • More
    • Follow
  • SUBSCRIBE

Marketing Land

Marketing Land
  • CMO
  • Social
  • SEM
  • SEO
  • Analytics
  • Display
  • Retail
  • MarTech
  • Resources
  • More
  • Home
  • Newsletters
  • Home
Analytics & Conversion

5 Keys To Improving Your A/B Tests

If you want valuable customer insights, you need to test — and test some more. Columnist Jordan Elkind offers five approaches for more effective marketing experiments.

Jordan Elkind on June 8, 2015 at 11:59 am
  • More
AB-test-ss-1920-800x450

As a marketer, you’re often running tests to find which subject line gets the most email opens, what homepage content improves your search engine optimization (SEO), what button color leads to more purchases, and so on.

Sometimes however, you need to answer a different type of question:

“What can I do to cause an incremental improvement versus doing nothing at all?”

Let’s look at an example. Say you wanted to run an email campaign with a 20 percent discount offer to customers who are likely to want to buy knitwear. To determine whether or not this campaign is successful, you ultimately need to know how many people made a purchase from this campaign that wouldn’t have done so if they never got the discount email.

If 80 people buy a new knit top, but almost all of those people would have done so anyway (except now they got a nice 20 percent off coupon), then you’ve actually hurt your company by giving discounts where they weren’t needed.

However, by properly designing and executing an experiment with a control group (or holdout sample) you are able to predict with great accuracy whether or not the campaign was effective. In short, you need to take a random subset of the population and do nothing to them, and then look at what they do compared to the group that received the treatment (an email campaign in this example).

While it can be hard to prioritize experiments, they can give you a better idea of the true return on investment (ROI) for your marketing, which can pay off big in the long run.

The following are five keys to effective marketing experiments.

1. Keep The Control And Test Groups Balanced

Stay disciplined about control groups — the temptation is there to shrink the control group down to maximize the earning potential of the test. The problem, however, is that without an adequately sized control group, deriving meaningful results and conclusions can become next to impossible since you don’t have a good read on baseline customer behavior.

Start out with a 50 percent control group, then cut it down as you begin to have a better sense of what works and what doesn’t.

2. Make Sure The Control And Test Groups Look The Same

Your control group needs to be the same as the test group. Not “kind of” — it needs to be identical. If you’re trying to see how a 10 percent discount works for mothers in Texas, then both your control and experimental groups need to be mothers from Texas.

To do this, you need to define your segment, and then randomly assign members to the test and control groups. Some email service providers can do this for you, otherwise you’ll need to use Excel or another tool to create the random groups yourself and then upload accordingly.

3. Eliminate Other Variables

Since you’re trying to see the impact of one idea, and only one idea, then you need to make sure that the test and control groups are otherwise identical in treatment. If one group gets a daily email about socks, then the other group needs to get that same sock-loving email; otherwise the results are tainted and therefore meaningless.

4. Keep It Simultaneous

Things change — strategies shift, seasons change, competitors rise and fall. Given these facts of business, you need to make sure that your test and control groups are being tracked at the same time. You can’t compare last month’s holdout group to this month’s test group.

5. Make Sure Results Are Trackable

The whole idea is to be able to measure the test and control groups against each other, so make sure you know what you’re tracking and how you’re going to track it.

And since the control group isn’t getting any emails, you can’t use clicks or opens to compare. (Customers can’t open an email they never received.) Look at revenue, orders, site visits, conversion rates, and so on during the test period to see whether the campaign did well enough to merit further investment.

To Recap

To measure the real impact of your next marketing campaign (and see its true incremental effect), make sure to have a holdout group that doesn’t receive the campaign so that you can properly compare results. Happy testing.


Opinions expressed in this article are those of the guest author and not necessarily Marketing Land. Staff authors are listed here.



About The Author

Jordan Elkind
Jordan Elkind heads the product team at Custora, an advanced customer analytics platform for e-commerce retailers. Prior to joining Custora, he earned an MBA from Wharton and worked in marketing analytics at Citi Cards.

Related Topics

AnalyticsAnalytics & Marketing ColumnChannel: Analytics & ConversionEmail MarketingHow To Guides: Email MarketingStatistics: Email Marketing

We're listening.

Have something to say about this article? Share it with us on Facebook, Twitter or our LinkedIn Group.

Get the daily newsletter digital marketers rely on.
See terms.

ATTEND OUR EVENTS

MarTech 2021: March 16-17

MarTech 2021: Sept. 14-15

MarTech 2020: Watch On-Demand

×

Attend MarTech - Click Here


Learn More About Our MarTech Events

February 23, 2021: SMX Report

April 13, 2021: SMX Create

May 18-19, 2021: SMX London

June 8-9, 2021: SMX Paris

June 15-16, 2021: SMX Advanced

August 17, 2021: SMX Convert

November 9-10, 2021: SMX Next

October 2021: SMX Advanced Europe

December17, 2021: SMX Code

Available On-Demand: SMX

×


Learn More About Our SMX Events

White Papers

  • A Beginner’s Guide to Omnichannel Marketing using Marketing Automation
  • The Top Five Objections That Hold Companies Back From Doing SEO
  • How To Optimize SEO With UGC
  • Email Tune-Up: A 5-Point Inspection to Get Your Program in Gear
  • Digital Marketing Report Q4 2020: Benchmarks and Insights for 2021
See More Whitepapers

Webinars

  • The Secret Behind SEO Success: Predict Rank with the Power of Data Science
  • How to Avoid the Digital Transformation Trap
  • How to Build a Marketing System of Record
See More Webinars

Research Reports

  • Local Marketing Solutions for Multi-Location Businesses
  • Enterprise Digital Asset Management Platforms
  • Identity Resolution Platforms
  • Customer Data Platforms
  • B2B Marketing Automation Platforms
  • Call Analytics Platforms
See More Research

h
Receive daily marketing news & analysis.
Marketing Land
Download the Marketing Land app on iTunes Download the Marketing Land App on Google Play

Channels

  • MarTech
  • CMO
  • Social
  • SEM
  • SEO
  • Mobile
  • Analytics
  • Retail
  • Display

Our Events

  • MarTech
  • SMX

Resources

  • White Papers
  • Research
  • Webinars
  • MarTech Conference
  • Search Marketing Expo

About

  • About Us
  • Contact
  • Privacy
  • Marketing Opportunities
  • Staff
  • Connect With Us

Follow Us

  • Facebook
  • Twitter
  • LinkedIn
  • Newsletters
  • Instagram
  • RSS
  • Youtube
  • iOS App
  • Google Play

© 2021 Third Door Media, Inc. All rights reserved.

Your privacy means the world to us. We share your personal information only when you give us explicit permission to do so, and confirm we have your permission each time. Learn more by viewing our privacy policy.Ok