FAQ: Everything Facebook has admitted about its measurement errors

To paraphrase J. Cole, Facebook messes up on its math one time; shame on Facebook. Facebook messes up on its math a second time; shame on those of us blindly trusting Facebook’s math. Facebook messes up on its math a third time; time to make a list.

To help myself and anyone else keep track of Facebook’s measurement errors, here’s an itemized list of each error Facebook has announced to date, a (hopefully) normal-speak explanation of what the error was, whether Facebook has corrected it or plans to, how it has/will correct the error and what impact the error had on the measurements Facebook reported.

Average watch time of Facebook page videos

When revealed? Late September 2016, revealed by The Wall Street Journal and addressed by Facebook the next day.

Error: Facebook calculated the average time people spent watching a video by dividing the cumulative time spent watching it by the number of views. But it only counted views that lasted at least three seconds, which set an artificial minimum for the measurement.

Corrected? Yes.

How was it corrected? Facebook has created two different replacements for the faulty average watch time metric.

The first, called “average watch time,” takes the total amount of time that people watched a video and divides it by the number of times that video was played, either automatically or after being clicked.

The second, called “unique average watch time,” takes the longest single instance that an individual watched a video and adds that duration — and only that duration — to the video’s total watch time, counts one play toward the total number of plays and divides total watch time by total number of plays.

Impact: The faulty metric overstated videos’ average watch time by between 60 percent and 80 percent.

Organic reach of Facebook Page posts: Part 1

When revealed? Mid-November 2016.

Error: When gauging the total number of people who saw at least one of a Page’s posts organically over a seven-day or 28-day period, Facebook didn’t de-duplicate that audience when reporting a Page’s organic reach in its Page Insights dashboard. In other words, if someone saw a Page’s post on Monday, and then another post from that Page on Tuesday, Facebook counted that person as two people.

Corrected? Yes. Facebook corrected the error in 2017.

How was it corrected? Facebook will count individuals exposed to a Page’s posts only one time per seven-day or 28-day period.

Impact: After the correction, Pages’ seven-day organic reach counts were expected to drop by 33 percent, on average, and their 28-day organic reach counts were expected to drop by by 55 percent, on average, according to Facebook.

Organic reach of Facebook Page posts: Part 2

When revealed? Mid-November 2016.

Error: When measuring Page posts’ organic reach, Facebook didn’t consider whether those organic posts actually appeared on someone’s screen.

Corrected? Not yet. Facebook expects to finalize the correction in early 2018; a Facebook spokesperson said the solution has taken longer to build than expected.

How will it be corrected? As with its ads, Facebook will no longer count a post as having reached someone until it appears on their screen.

Impact: Once the correction rolls out, Facebook expects Pages’ organic reach counts to drop by 20 percent, on average.

Facebook video ad completion rate

When revealed? Mid-November 2016.

Error: Sometimes Facebook streams a video ad for “a fraction of a second” longer or shorter than the video’s actual length because the video’s visual and audio don’t line up, leading it to mis-time when the end of a video has been watched.

Corrected? Yes.

How was it corrected? Facebook has updated “how we read the video length,” according to the company. It’s worth pointing out that this metric isn’t truly a measure of a video ad’s completion rate. It’s a measure of how many times the end of a video was viewed, including times when someone skipped to the end of the video and only played that part (which would be a way to juice the measurement).

Impact: Post-correction, Facebook thinks advertisers will see a 35 percent increase in the number of “video watches at 100%,” which is not necessarily the same thing as completed views.

Average time spent on Facebook Instant Articles

When revealed? Mid-November 2016.

Error: Facebook was supposed to calculate the amount of time people spent reading an Instant Article by dividing the total time an Instant Article’s entire audience spent reading by the total number of times the Instant Article was opened. But it didn’t. Instead it took an average from a histogram it made plotting time spent.

Corrected? Yes.

How was it corrected? Facebook now calculates average time spent the way it was supposed to calculate it all along.

Impact: Prior to the correction, the average time spent measured on Instant Articles was, on average, seven to eight percent higher than it should have been, according to Facebook.

Referral traffic from Facebook to websites & mobile apps

When revealed? Mid-November 2016.

Error: The “referrals” metric in Facebook’s Analytics for Apps — an analytics tool Facebook offers for third-party mobile apps and websites — was supposed to report how many times people clicked on a Facebook post to open a business’s mobile app or visit its website. It was supposed to report only that number. But it didn’t.

Given that those posts linking to a business’s mobile app or site could also include a Facebook-native photo or video, Facebook incorrectly counted clicks on those photos or videos as referral clicks, even though these clicks didn’t send anyone to the business’s app or site but only enlarged the photo or video.

Corrected? Yes. Facebook corrected the error in 2017.

How was it corrected? Facebook will limit its count to times when people clicked a post and actually opened the corresponding app or site.

Impact: Facebook incorrectly counted, on average, 30 percent of the referrals it reported to mobile apps and websites using its Analytics for Apps tool, according to the company.

Like, Share and Comment Counts for Off-Facebook Links

When revealed? Discovered by Marketing Land in November 2016 and confirmed by Facebook in December 2016.

Error: Facebook’s Graph API can report the total number of engagements (shares, likes and comments) a link received on or off Facebook, including when people paste the link into a Facebook status update and when they click the Facebook like button embedded on the link’s actual web page. Facebook’s mobile search results can report a subset of the Graph API total, specifically the number of shares and comments on Facebook. But sometimes the mobile search number is larger than the Graph API number, which should never be the case.

Corrected? Yes. Facebook corrected the error in 2017.

How was it corrected? From a company blog post that Facebook updated on June 9, 2017: “We identified a logging issue affecting the Graph API and metrics for the Like and Share buttons. This caused a discrepancy between data shown on the buttons and seen in mobile search query results. We are fixing this today to ensure consistent reporting between the Graph API, Like / Share buttons, and mobile search query results. Going forward, some high traffic publishers will see an increase in their engagement metrics. This fix applies to new URLs that are shared to Facebook and will not change metrics shared before June 9, 2017.”

Impact: Unclear. For one NBCNews.com link I examined in late November 2016, the mobile search number was 71 percent larger than the Graph API number.

Reaction Breakdowns of Facebook Page Live Videos

When revealed? Early December 2016.

Error: Facebook’s Page Insights dashboard breaks down the number of “reactions” a Page’s Live video generated by whether the reactions happened while people were watching the live stream — “reactions on post” — or whether they were received on others’ posts that included the Live video — “reaction on shares” — such as when someone shares, or Facebook-retweeted, a Page’s Live video post to their friends and those friends clicked the like/reaction button on the friend’s post after the live stream had already ended.

The “reactions on post” number was supposed to only measure one reaction per individual live viewer, and it did. But people may react multiple times to a live video, such as by clicking “haha” each time the stream makes them laugh. Facebook incorrectly included those extra reactions to the “reactions on shares” figure, even though they weren’t reactions on shared posts.

Corrected? Yes. Facebook corrected the error in mid-December 2016.

How was it corrected? Facebook will report the total number of reactions generated during the live stream within the “reactions on post” metric.

Impact: The corrections will result in a 500 percent average increase in the number of “reactions on post” and a 25 percent average decrease in the number of “reactions on shares,” according to Facebook.

iPhone Traffic for Instant Articles

When revealed? Mid-December 2016.

Error: Facebook undercounted iPhone traffic for publishers who had been using Facebook’s legacy ComScore integration and had updated their sites to the more secure HTTPS protocol.

Corrected? Yes.

How was it corrected? Facebook has fixed the issue for Instant Articles traffic going forward and is working with ComScore to come up with corrected traffic estimates for the affected period.

Impact: Facebook has not said what the impact was. According to The Wall Street Journal, for most publishers, less than 1 percent of traffic was affected, but some saw 10 percent to 20 percent of traffic affected.

Video Carousel Ad Link Clicks

When revealed? May 2017.

Error: For a year, Facebook mischarged some advertisers for clicks on video carousel ads. Facebook was only supposed to bill these advertisers when people clicked on a link to the advertiser’s site, but it mistakenly also charged them when smartphone users on Facebook’s mobile site clicked to play the video at full screen.

Corrected? Yes.

How was it corrected? Facebook has refunded the affected advertisers in full.

Impact: Of all ad impressions served by Facebook in the year the error was in effect, 0.04 percent of impressions were affected, and the median amount of money refunded to an advertiser was $10.

Mobile Web Video Views

When revealed? November 2017.

Error: Videos loaded on Facebook’s mobile site continued to play even after they were scrolled out of view, leading Facebook to charge advertisers for the background views.

Corrected? Yes.

How is it being corrected? Facebook has refunded the affected advertisers in full.

Impact: The error is said to be smaller than the carousel ad link click error disclosed in May 2017, with a couple hundred advertisers being refunded more than $5,000.

Instant Article Video Views

When revealed? November 2017.

Error: Facebook mischarged advertisers for videos that were played out of view. The error affected videos embedded within Instant Articles that were accessed through Facebook’s Android app over a slow internet connection.

Corrected? Yes.

How is it being corrected? Facebook has refunded the affected advertisers in full.

Impact: The error is said to be smaller than the carousel ad link click error disclosed in May 2017, with a couple hundred advertisers being refunded more than $5,000.


About The Author

Tim Peterson
Tim Peterson, Third Door Media's Social Media Reporter, has been covering the digital marketing industry since 2011. He has reported for Advertising Age, Adweek and Direct Marketing News. A born-and-raised Angeleno who graduated from New York University, he currently lives in Los Angeles. He has broken stories on Snapchat's ad plans, Hulu founding CEO Jason Kilar's attempt to take on YouTube and the assemblage of Amazon's ad-tech stack; analyzed YouTube's programming strategy, Facebook's ad-tech ambitions and ad blocking's rise; and documented digital video's biggest annual event VidCon, BuzzFeed's branded video production process and Snapchat Discover's ad load six months after launch. He has also developed tools to monitor brands' early adoption of live-streaming apps, compare Yahoo's and Google's search designs and examine the NFL's YouTube and Facebook video strategies.