Is Google Letting Anyone Do “Scientific” Surveys A Good Thing?

Last week, Google rolled out its new Google Consumer Surveys service. Finally, “statistically significant, valid results at scale” at a price anyone can afford — only $0.10 per response! But is letting the untrained masses create “scientific” polls dangerous?

I’m not a professional pollster, nor have I had training in creating polls. I’d be very curious on what those who do polls for a living think about the service, and I might look into that more later. I imagine some will talk it down because it does, after all, potentially undercut the expensive opinion polls out there.

I am, however, a veteran of having to deal with polling data as a journalist for over 20 years. I’ve seen all types of problems with polls. The biggest problem isn’t that they weren’t done “scientifically” but rather in how they posed questions. That’s something that the new Google service doesn’t fix.

Consider the poll I ran and wrote about today in Survey: Nearly 80% Trust Google As Much Or More Than A Year Ago. I asked, ”Do you trust Google as much as a year ago?” That could have been posed as a purely yes/no question. Had I done that, I wouldn’t have detected that the vast majority of people actually say they trust Google the same as a year ago. The results would be massively different — and they were. I know this, because out of curiosity, I ran the question again as a yes/no.

What if I’d asked something like “Is Google Evil?” Charging the question up in that way could have given me different results.

The press already has a tough enough time dealing with “professional” polls, interpreting what they accurately show or subjecting them to proper critical analysis. Now we potential face a flood of new polls from people with no training at all.

It’s not like non-professionals haven’t been able to run polls. There’s no lack of ways to do this. The Washington Post ran one in January showing that 65% of people were going to close their Google Accounts over its announced privacy policy change.

That didn’t happen, obviously. The saving grace for the Washington Post is that the fine print of the poll said:

This is a non-scientific user poll. Results are not statistically valid and cannot be assumed to reflect the views of Washington Post users as a group or the general population.

Given all those disclaimers, you wonder why they bothered to run it at all. But today, it could easily run a “scientific” poll using Google’s service and get data that’s designed to be reflective of the internet population in the United States, data that Google even backs as accurate with a white paper.

End result? Potentially we get a lot of bad polls that have the appearance of being accurate.

I put some of these concerns to Paul McDonald, who is the senior product manager for Google Consumer Surveys. Here’s our email exchange:

Danny: The service comes across as allowing anyone to do scientific polling. But if you’re not careful about how you ask a question, you’re going to get skew. If you don’t think carefully about the questions themselves, you get skew.

Paul: I think your concerns about the quality of the data from self-service survey platforms are well known in the research community, as the mantra goes “Garbage in, garbage out”, though I wouldn’t characterize it as “dangerous.”

We try to encourage survey best practices in our help contentprogram policies and by providing survey templates to guide new researchers. For example leading questions, push-polling and irrelevant survey text are all prohibited.

We also use statistical methods to ensure that viewers of the survey data understand the margin of error and potential bias introduced by our sample via the error bars on the charts and sample bias table at the bottom of the report.

In the end we are providing a platform which can be used to create professional and statistically accurate surveys. We believe users who view these reports can spot questions that will lead to a biased result and will make a call on the effectiveness of the survey creator.

Danny: The polling method seems scientific in nature, but if you ask bad questions, or if you don’t know how to interpret the results, ugh. I don’t know of any other self-serve polling system this inexpensive that allows this type of scientific surveying.

Paul: I hear you, I guess my issue is with your positioning. You seem to be saying that Google Consumer Surveys is more responsible for this than any other survey platform. There are definitely things we can do, some of which we’ve already addressed, to help researchers create good surveys but as you mentioned even scientific surveys done by professionals can have issues. One of our value propositions is that with our platform you can hand a link to our reports and other people can analyze the results, data and question themselves.

I do love that Google Consumer Surveys allows for anyone to share their full data, though I’m hoping this will be improved by also adding some guidance for attribution, so that if people link back to the survey, you understand the organization that ran the poll and give proper credit.

I also love the other type of polling that the product seems especially aimed at, helping marketers get back consumer reactions to product changes, needs, desires, logos and more. Be sure to see the case studies page and the how it works page for examples of this.

As for my worries about broader opinion polls, perhaps the new service may help there in promoting more critical thinking about polls in general. Even the “scientific” polls from trained professionals often fail to measure up, especially if those professionals have deliberately constructed a poll to reach a particular conclusion. If there’s a flood of new polls that develop, all polls might get held to a higher standard.

Related Topics: Channel: Strategy | Features & Analysis | Google: Consumer Surveys


About The Author: is Founding Editor of Marketing Land. He’s a widely cited authority on search marketing and internet marketing issues, who has covered the space since 1996. Danny also serves as Chief Content Officer for Third Door Media, which publishes Search Engine Land and produces the SMX: Search Marketing Expo conference series. He has a personal blog called Daggle (and keeps his disclosures page there). He can be found on Facebook, Google + and microblogs on Twitter as @dannysullivan.

Connect with the author via: Email | Twitter | Google+ | LinkedIn

Marketing Day:

Get the top marketing stories daily!  


Other ways to share:

Read before commenting! We welcome constructive comments and allow any that meet our common sense criteria. This means being respectful and polite to others. It means providing helpful information that contributes to a story or discussion. It means leaving links only that substantially add further to a discussion. Comments using foul language, being disrespectful to others or otherwise violating what we believe are common sense standards of discussion will be deleted. You can read more about our comments policy here.
  • Anonymous

    I did a search yesterday on ’causes of cancer’ and the results were pretty much as you’d expect. EVERYTHING causes cancer.
    If you’re going to criticize Google for allowing ‘anyone’ to take an opinion poll, perhaps you should judge EVERYONE for their bias and assumptions in every single publication ever written.

    Scientists have no idea what causes cancer, but release new so called ‘findings’ daily. Educated people know these polls along with other polls and studies are not accurate. Saying Googles platform has some inherent weakness other methods don’t doesn’t really sit well with me.

  • Brosix

    I think this could be a good way to get general information, but should probably be taken with a grain of salt.

  • Ellie K

    I wish that “scientists” or rather, the scientific press WOULDN’T release so many new findings so frequently. It overwhelms me. I don’t know enough to determine what is significant, what is noise. I was a statistician for a Dept of Health Services for a large state in the U.S., father and uncle are M.D.’s, yet I still feel this way! These aren’t problems in research necessarily. But how to tell what is a preliminary finding versus final (and in many fields of medicine)? That’s to much to expect from a few overworked science journalists.

  • Ellie K

    Is this a good thing, for Google to encourage anyone, everyone to do surveys? No, not really. Or better, they shouldn’t describe them as scientific surveys. Those were Google’s words, not @dannysullivan:twitter ’s. 
    Surveys aren’t easy to do correctly. I worked as a statistician for 15 years, mostly with IBM storage and later, financial regulation & fraud detection. I never did a survey though. I was assigned to do one recently, for a state health dept. I thought it would be easy. I was wrong. The subject matter was basic, customer service and delivery timeliness. It wasn’t complicated statistically, but there were many practical things: How to phrase questions, order to ask them, which responses should be excluded and why. I needed guidance from my co-workers with experience in writing survey questions, how they flowed, even help interpreting the results! I’m doubtful about the Google survey product, it might be good for some, but a waste of time for others.

Get Our News, Everywhere!

Daily Email:

Follow Marketing Land on Twitter @marketingland Like Marketing Land on Facebook Follow Marketing Land on Google+ Subscribe to Our Feed! Join our LinkedIn Group Check out our Tumblr! See us on Pinterest


Click to watch SMX conference video

Join us at one of our SMX or MarTech events:

United States


Australia & China

Learn more about: SMX | MarTech

Free Daily Marketing News!

Marketing Day is a once-per-day newsletter update - sign up below and get the news delivered to you!