The digital marketing landscape is moving faster than ever before, with dozens of new tools, platforms, and advertising media coming into existence every year. When making a decision to adopt one tool over another or making an entrance into a new market, the case study oftentimes becomes an important factor in whether the trigger is pulled.case study

While a proven tool or medium is certainly less risky to adopt compared to one that is completely unproven, it’s worth remembering that case studies are generally crafted to make the object look as good as possible. The results presented may be distortions of reality, and any single study must be taken with a grain of salt.

In this article, I hope to shed some light on how to effectively review a case study — including what questions to ask yourself or others while reading them.

Case Study Result Metrics

Some case studies contain little more than the fact that a particular marketer was happy with the results of using the certain tool or market. Though such case studies help in a manner similar to product reviews on retail websites, marketers are increasingly demanding hard numbers to quantify the potential gain.

The most commonly cited quantitative results are the following:

  • Lower CPC or CPM (cost per click/mille)
  • Higher CTR (click-through rate)
  • Improved conversion rate
  • Reduction in CPA (cost per action)
  • Improvement in ROI/ROAS (return on investment/advertising spend)
  • Increase in amount of conversions, revenue, or other KPI

While all of these are positive results, some are more important than others, and some can be misleading. The questions below will help cut through smoke and mirrors to help determine how applicable the case study is to a marketer’s business goals.

The questions are broadly classified into three categories: high-level applicability questions, experimental design questions, and results validation.

Applicability

1. Does the subject of the case study have a similar business model and KPI as mine?

This is an obvious but important question. A tool or marketplace that works for one industry may not work for another, depending on the target audience and creative format. If the subject of the case study is in a similar vertical to yours with similar business goals, then the relevance of the study will improve.

2. When was the case study written, and is it still applicable now?

In today’s rapidly evolving digital landscape, case studies that are even two or three years old may become outdated, since the tool features, or market user base and competition, may have changed significantly since then. This is particularly applicable to relatively new markets such as social or mobile.

Experimental Design

3. Was this a simultaneous or sequential test?

Simultaneous tests are generally better than sequential, because seasonality or market changes may have affected results during the test period. However, simultaneous tests also have the challenge of selecting campaigns that are sufficiently similar to each other (an example of control factors, detailed in #6).

4. What was the sample size (number of impressions/clicks/conversions)?

If the sample size of the comparison is too small, the results do not mean much. For example, if 2 conversions occurred for one scenario and 3 for the other, this would be a 50% increase in conversions on paper, but it could have easily happened purely by chance alone.

5. How long did the test run for, and what periods were used in the comparison?

Similar to sample size, a test that is too short has greater potential for noise. However, a test that runs for too long introduces potential for various confounding factors that may have influenced test results (especially in a sequential test).

6. What kind of control factors (variables held constant) were used in the test?

For the comparison to be as fair as possible, it is vital for there to be controls put into place. Spend, for example, is one factor that absolutely needs to have been controlled for in the test, since it is easy to get higher efficiency with a lower budget.

7. Was the result statistically significant?

Related to #4, if the case study was based on too little data, a seemingly good result can be obtained just by chance. Statistical significance will put this doubt to rest; if the subject of the case study can respond to this question with a definitive yes, then there is greater reason to believe that the study was grounded on good data and science.

Results Validation

8. How about the results for [insert metric]?

I rarely see A/B tests where all metrics point towards a positive result. For instance, CPA may have improved, but conversion rate may have gone down (which means the CPA improvement was driven by CPC/CPM decrease). These negative or neutral components of the result are sometimes withheld from print. Particularly concerning is when results of higher-funnel metrics such as CPC or CPM are given, while lower-funnel metrics such as CPA and ROI are not.

9. How do I know that the CPC/CPM will not rise in the future?

Since price is influenced by competitive landscape, there is risk that an early-stage market will become increasingly more expensive in the future. Early adoption of new markets may pay off while the market is still fresh, but results may become stagnant with competition increase.

Conclusions And Tips For Marketers

With data-driven marketing becoming the new norm, case studies are also become increasingly quantified. While there are many metrics that can be presented to seemingly show a positive result, only a small subset may actually be applicable to any single marketer.

The above questions should help shed light on how much to value the results of a case study for a tool, platform, or marketplace.

 (Stock image via Shutterstock.com. Used under license.)

Opinions expressed in the article are those of the guest author and not necessarily Marketing Land.

Related Topics: Analytics | Analytics & Marketing Column | Channel: Analytics | How To Guides | How To Guides: Marketing Analytics | Internet Marketing

Sponsored


About The Author: is a Senior Business Analyst in the Digital Marketing team at Adobe, conducting market research across multiple advertising channels including search, display, and social media.



Sign Up To Get This Newsletter Via Email:  


Share

Other ways to share:

Read before commenting! We welcome constructive comments and allow any that meet our common sense criteria. This means being respectful and polite to others. It means providing helpful information that contributes to a story or discussion. It means leaving links only that substantially add further to a discussion. Comments using foul language, being disrespectful to others or otherwise violating what we believe are common sense standards of discussion will be deleted. You can read more about our comments policy here.
  • Trevor Sumner

    I don’t see much about marketing. Just advertising and assuming all interactions are equal. Is this a media planner’s guide to the world?

  • Kohki Yamaguchi

    Touche, the metrics I used was from the paid media side. But most of the general conclusions and pointers should be applicable for all marketing case studies.

  • Jane

    Hi Kohki, I agree with your conclusion on Case studies. The pace at which things change in digital marketing is breakneck. The strategist is the number cruncher, the person who calls all the individual experts to account at the end of the month to see if they are increasing or reducing ROI. It is difficult to over-state the importance of this role.
    Reference:- http://www.dpfoc.com/australia/blog/successful-digital-marketing-is-like-beautiful-music

  • http://www.acewebacademy.com/ Acewebacademy

    It is extremely easy to get misled by so called
    case studies as you work
    on a marketing strategy- Mighty useful tips there.

 

Get Our News, Everywhere!

Daily Email:

Follow Marketing Land on Twitter @marketingland Like Marketing Land on Facebook Follow Marketing Land on Google+ Subscribe to Our Feed! Join our LinkedIn Group Check out our Tumblr! See us on Pinterest

 
 

Click to watch SMX conference video

Join us at one of our SMX or MarTech events:

United States

Europe

Australia & China

Learn more about: SMX | MarTech


Free Daily Marketing News!

Marketing Day is a once-per-day newsletter update - sign up below and get the news delivered to you!