• Marketing Land
  • Sections
    • CMO
    • Social
    • SEM
    • SEO
    • Analytics
    • Display
    • Retail
    • MarTech
    • Resources
    • More
    • Home
  • Follow Us
    • Follow
  • Marketing Land
  • CMO
  • Social
  • SEM
  • SEO
  • Analytics
  • Display
  • Retail
  • MarTech
  • Resources
  • More
    • Follow
  • SUBSCRIBE

Marketing Land

Marketing Land
  • CMO
  • Social
  • SEM
  • SEO
  • Analytics
  • Display
  • Retail
  • MarTech
  • Resources
  • More
  • Home
  • Newsletters
  • Home
Analytics & Conversion

Achieve big testing wins by analyzing results by device type

Responsive design sites utilize the same HTML for users, regardless of device -- but that doesn't mean that all users are getting the same experience. Columnist Abraham Nord shares a case study in which segmenting results by device type provided insights that led to a significant conversion boost.

Abraham Nord on September 8, 2017 at 1:13 pm
  • More

For some time now, responsive design has been all the rage — and this trend shows no sign of letting up as the dominant configuration for mobile websites.

Responsive design allows us to present a consistent site experience to users regardless of screen size. Because the HTML is the same for all devices, it has the added benefit of also reducing development and maintenance time. However, different experiences based on screen size or device type can pose challenges when running tests and analyzing results.

Testing challenges

For our discussion, we will consider the three most common device types that visitors use when accessing a website:

  1. Mobile
  2. Tablet
  3. Desktop

Even with responsive design, the user experience for a website can be radically different for each of these devices. Running a test without considering and analyzing results specific to each device type can lead to erroneous conclusions, resulting in poor decisions that could have a potentially disastrous impact on conversion rates.

Understand test results by device type

Many testing tools allow us to segment results by device type in order to get a more granular picture of performance on our website. However, depending on how your responsive site is coded, this may not be the best indicator of the visitor’s experience.

For example, our landing pages are coded responsively based on the width of the visitor’s browser. So, if a visitor has their browser open on only half of their desktop screen, they would actually be experiencing the “tablet” version of our page, even though their device type is still “desktop.”

Understanding how pages behave and how devices are labeled is critical for accurate analysis of your testing data.

Test example

Let’s finish up by looking at a real-world example to show the dramatic difference this can make on the performance of a website or landing page.

We recently ran an A/B/C test of a new look and feel for a responsive landing page. After about six weeks, both test variations were outperforming the control (by 51.79 percent and 51.23 percent, respectively) for all traffic. We could say with statistical confidence that both test variations were an improvement over the control; however, there was no statistically significant difference between test variations to help us determine which one to use.

We could have simply run the test longer until statistical confidence was achieved between test variations, but before doing so, we decided to look at the results by browser screen width (described as mobile, tablet and desktop below). What we found was startling.

  • Test Variation 1 actually had a 41 percent decrease in conversions for mobile visitors, but a 148.03 percent increase in conversions for desktop visitors.
  • Test Variation 2 had a 42 percent increase in conversions for mobile visitors and a 62.64 percent increase in conversions for desktop visitors.
  • Tablet data for both variations was too small for statistically relevant results.

Our conclusion

Based on this data, we concluded the test with a hybrid winning page, combining the desktop experience of Variation 1 and the mobile experience of Variation 2.

This hybrid approach resulted in an overall 86.40 percent increase in conversions instead of the ~51 percent increase that we were looking at previously. The hybrid solution also allowed us to conclude the test sooner than would have otherwise been possible.

I recommend that testers analyze results by device type or screen width whenever possible and implement hybrid solutions utilizing the best results for each experience. The results may surprise you!


Opinions expressed in this article are those of the guest author and not necessarily Marketing Land. Staff authors are listed here.



About The Author

Abraham Nord
As head of the Conversion Improvement department at SmartSearch Marketing, Abraham leads landing page testing strategy, implementation and analysis along with client communication and delivery. Abraham brings 8+ years of experience in conversion improvement, landing page testing and usability with a specific focus towards B2B and business with a complex sale. Prior to joining SmartSearch Marketing, Abraham successfully ran his own ecommerce site and holds a bachelor of fine arts in painting along with a degree in web design and interactive media. Abraham currently resides in Denver, CO and enjoys spending time with his family, camping, hiking, playing tennis and online gaming.

Related Topics

AnalyticsChannel: Analytics & ConversionConversion Rate OptimizationDesign, Usability & Conversion Column

We're listening.

Have something to say about this article? Share it with us on Facebook, Twitter or our LinkedIn Group.

Get the daily newsletter digital marketers rely on.
See terms.

ATTEND OUR EVENTS

MarTech 2021: March 16-17

MarTech 2021: Sept. 14-15

MarTech 2020: Watch On-Demand

×

Attend MarTech - Click Here


Learn More About Our MarTech Events

February 23, 2021: SMX Report

April 13, 2021: SMX Create

May 18-19, 2021: SMX London

June 8-9, 2021: SMX Paris

June 15-16, 2021: SMX Advanced

August 17, 2021: SMX Convert

November 9-10, 2021: SMX Next

October 2021: SMX Advanced Europe

December17, 2021: SMX Code

Available On-Demand: SMX

×


Learn More About Our SMX Events

White Papers

  • The State of Local Marketing Report 2020-2021
  • Quality CRM Data: The Key to Delivering Great Customer Experiences
  • How the Microsoft Search Network Can Maximize Your Search Campaigns
  • The Marketer’s Playbook for Customer Acquisition
  • How To Optimize SEO With UGC
See More Whitepapers

Webinars

  • How to Avoid the Digital Transformation Trap
  • How to Build a Marketing System of Record
  • Meet BIMI: The brand-boosting email security marketers must have for 2021
See More Webinars

Research Reports

  • Local Marketing Solutions for Multi-Location Businesses
  • Enterprise Digital Asset Management Platforms
  • Identity Resolution Platforms
  • Customer Data Platforms
  • B2B Marketing Automation Platforms
  • Call Analytics Platforms
See More Research

h
Receive daily marketing news & analysis.
Marketing Land
Download the Marketing Land app on iTunes Download the Marketing Land App on Google Play

Channels

  • MarTech
  • CMO
  • Social
  • SEM
  • SEO
  • Mobile
  • Analytics
  • Retail
  • Display

Our Events

  • MarTech
  • SMX

Resources

  • White Papers
  • Research
  • Webinars
  • MarTech Conference
  • Search Marketing Expo

About

  • About Us
  • Contact
  • Privacy
  • Marketing Opportunities
  • Staff
  • Connect With Us

Follow Us

  • Facebook
  • Twitter
  • LinkedIn
  • Newsletters
  • Instagram
  • RSS
  • Youtube
  • iOS App
  • Google Play

© 2021 Third Door Media, Inc. All rights reserved.

Your privacy means the world to us. We share your personal information only when you give us explicit permission to do so, and confirm we have your permission each time. Learn more by viewing our privacy policy.Ok