• http://durak.org/sean/ sed

    This seems like a weak approach to fixing a problem that may not exist. Google analytics already includes several effective hurdles to prevent robot traffic from being recorded. the IAB list of blacklisted user agents is closed source, so you never know who you are excluding. The User-Agent client http header field is optional and forge-able, so no evil botmaker is going to be thwarted at this level.

  • http://www.marketingtechblog.com Douglas Karr

    Any idea if Semalt is on that list? They wreak havoc with virtually every client I have.

  • anilopez

    1) “Google analytics already includes several effective hurdles to prevent robot traffic from being recorded” How do you know that?
    2) “IAB list of blacklisted user agents is closed source” No, it is not. It is posted out there.
    3) There are tones of bots but most of them cannot execute JS so most of them don’t impact GA numbers.

  • http://durak.org/sean/ sed

    hi @anilopez:disqus, robot traffic is less likely to be recorded in Google Analytics because it requires the user agent to overcome the hurdles of javascript, image requests, and cookies which (as you note in your own comment) few spiders are programmed to do. the IAB/ABC International Spiders & Bots List is available to (paying) members only: http://www.iab.net/1418/spiders – if this resource is legally available somewhere for free, please share the location! -sean

  • anilopez
  • Nataliya Khachaturyan

    Let me explain why Semalt has negative response. Daily our technical robots visit many websites. These robots harvest statistical data for our service and don’t cause any harm to the users’ web resources.
    We understand each user who asks us to disable crawling activity on their
    websites. We bring apologies to each user and honor the request. That’s why we’ve created a special tool to remove sites from the list of web resources we visit – Semalt Crawler http://bit.ly/1gorS73
    Nevertheless, many people, many minds. Some people show understanding to these actions but some cannot hide their emotions.
    If you have questions regarding our service, I will be glad to answer them.