Webtrends Optimize tries to identify bot traffic, flag it to you, and give you a mechanism to filter it out.
It is worth noting that we do not generate this traffic. It is coming to your website, and our goal is to help you identify that it happened and make it as easy as possible to remove, thus giving you more meaningful test data to analyse - helping you make decisions with less noise influencing the data.
How we identify bot traffic
We use an often-updating list of bot IP user agents. We currently cover thousands of unique bot/crawler services and countless versions of each.
Examples:
We attempt to identify countless crawlers, scrapers, bots, etc., from companies both big and small, both at overt layers e.g. identified as Google/Bing/Yahoo/Yandex, and less overt layers e.g. code run in a Google Apps script.
Examples of this:
Bots from: Ahrefs, bing, yandec, dotbot, sitescore, deepcrawl, cocolye, smtbot, pingbot, pingdom, twitter, grmamarly, semrun, etc.
Large Language Model providers: OpenAI, Claude, Perplexity, etc.
Google services: Youtube links, search console, adwords, page speed insights, earth, toolbar, apps-script, keyword suggestions, image search, etc.
Yahoo services
Troubleshooting bot traffic
If you believe a particular bot is spamming your test, we recommend you:
Go to Full Reporting for that experience
Extract the data. Be sure to include the fields Device and User Agent.
Download the extract
Filter the data in Excel or similar to Device is Bot.
Analyse interesting parts of the user agent to see which specific bot is spamming your test.
How we flag bot traffic to you
When you click into any given experience, you'll see a summary of traffic flowing in. Bot traffic will be flagged to you there:
How you can filter out bot traffic from your reports
You can filter out this traffic in reporting very easily:
