Skip to main content

Aborting the tag

How and when you choose for Webtrends Optimize not to run

O
Written by Optimize Team

There are many reasons where you could choose for Optimize to not run. This document covers how, where, and a few examples.

How and where

These are all Javascript conditions. We recommend making sure the information is available before the wt.js tag runs on the page, so that we're not waiting for the page to load prior to aborting the tag.

  • If this is possible, the code you'll write should go into the pre-init/global tag script section of the tag confguration.

  • If this is not possible, the code should go into the post-load/tag trigger script section, where you can poll for some conditions before running WT.optimize.setup, and selectively abort at that point. Note though, that polling for conditions slows down page load times.

Notes:

  • Aborting the tag does not force the rest of the code to stop. You may wish to either use throw, return or if conditions to make sure future parts of the tag do not run (if needed).

Examples

Abort the tag for known bots

You may not want known bots to run Optimize - aborting the tag for these will save some of your session usage.

var botregex = /(AddSearchBot|AI2Bot|AI2Bot-DeepResearchEval|Ai2Bot-Dolma|aiHitBot|amazon-kendra|Amazonbot|AmazonBuyForMe|Amzn-SearchBot|Amzn-User|Andibot|Anomura|anthropic-ai|ApifyBot|ApifyWebsiteContentCrawler|Applebot|Applebot-Extended|atlassian-bot|Awario|AzureAI-SearchBot|bedrockbot|bigsur.ai|Bravebot|Brightbot 1.0|BuddyBot|Bytespider|CCBot|Channel3Bot|ChatGLM-Spider|ChatGPT Agent|ChatGPT-User|Claude-SearchBot|Claude-User|Claude-Web|ClaudeBot|lightpanda|Cloudflare-AutoRAG|CloudVertexBot|cohere-ai|cohere-training-data-crawler|Cotoyogi|Crawl4AI|Crawlspace|Datenbank Crawler|DeepSeekBot|Devin|Diffbot|DuckAssistBot|Echobot Bot|EchoboxBot|ExaBot|FacebookBot|facebookexternalhit|Factset_spyderbot|FirecrawlAgent|FriendlyCrawler|Gemini-Deep-Research|Google-CloudVertexBot|Google-Extended|Google-Firebase|Google-NotebookLM|GoogleAgent-Mariner|GoogleOther|GoogleOther-Image|GoogleOther-Video|GPTBot|iAskBot|iaskspider|iaskspider\/2.0|IbouBot|ICC-Crawler|ImagesiftBot|imageSpider|img2dataset|ISSCyberRiskCrawler|kagi-fetcher|Kangaroo Bot|KlaviyoAIBot|KunatoCrawler|laion-huggingface-processor|LAIONDownloader|LCC|LinerBot|Linguee Bot|LinkupBot|Manus-User|Meta-ExternalAgent|Meta-ExternalFetcher|meta-webindexer|MistralAI-User|MistralAI-User\/1.0|MyCentralAIScraperBot|netEstate Imprint Crawler|NotebookLM|NovaAct|OAI-SearchBot|omgili|omgilibot|OpenAI|Operator|PanguBot|Panscient|panscient.com|Perplexity-User|PerplexityBot|PetalBot|PhindBot|Poggio-Citations|Poseidon Research Crawler|QualifiedBot|QuillBot|quillbot.com|RuxitSynthetic|SBIntuitionsBot|Scrapy|SemrushBot-OCOB|SemrushBot-SWA|ShapBot|Sidetrade indexer bot|Spider|TavilyBot|TerraCotta|Thinkbot|TikTokSpider|Timpibot|TwinAgent|VelenPublicWebCrawler|WARDBot|Webzio-Extended|wpbot|WRTNBot|YaK|YandexAdditional|YandexAdditionalBot|YouBot|ZanistaBot|Yahoo Link Preview|Yahoo! Slurp|YOURLS|webcrawler|Crawler Bot|Youtube-Links|Google Search Console|Google-Adwords|Google Page Speed Insights|GoogleEarth|GoogleToolbar|Google-Apps-Script|Google Keyword Suggestion|Google-SearchByImage|Yahoo Ad monitoring|rssowl|yandex.com\/bots|GetIntent Crawler|grammarly|feedly|web spider|Go-http-client|Mediapartners-Google|AdsBot|python-requests|Google\s?bot|AhrefsBot|abevalbot|bingbot|YandexBot|DotBot|sitescorebot|deepcrawl|lumar|similarweb|cocolyzebot|SMTBot|sabot|pingbot|leavemealonebot|pingdom.com|onetrustbot|storebot|getthit.com|taboolabot|radius compliance bot|klarnabot|pricewatcher|moatbot|Baiduspider|GrapeshotCrawler|AlphaBot|proximic|coccocbot|NetcraftSurveyAgent|SemrushBot|seekportbot|Mail.RU_Bot|YandexImages|ZmEu|BingPreview|FeedDemon|Twitterbot|SimplePie|Webspider|CloudFlare-AlwaysOnline|MagpieRSS|longurl|tt-rss|Google Favicon|headlesschrome|GroupHigh|hanaleibot|ia_archiver|google.com\/feedfetcher|MJ12bot|archive.org_bot|Yeti\/|zgrab|vkShare|Screaming Frog SEO|klaviyofootprintscanner|NetSeer crawler|crawler4j|letsencrypt.org|Google-Site-Verification|DomainSONOCrawler|Download Master|WhatsApp\/\d|GnuTLS|Cliqzbot|riddler.io\/|okhttp|filterdb.iss.net\/crawler|Google Web Preview|AppEngine-Google|Jersey\/|DuckDuckGo-Favicons-Bot|rogerbot|YahooCacheSystem|Apache-HttpClient|bidswitchbot\/|Clickagy Intelligence Bot|MaxPointCrawler\/|LightspeedSystemsCrawler|Photon\/\d|FeedBurner\/|Wappalyzer|LinkpadBot|Crawler\/\d|BrokenLinkCheck.com|libcurl\/|TurnitinBot|Microsoft Office Protocol Discovery|Microsoft Windows Network Diagnostics|skype-vhod|\+https?:\/\/[\w\.\-\/]+(robot|bot|crawler|embed|fetcher|seznambot|feedparser|daum.net|brandwatch|spider|checksite|google.com|goo.gl\/))/i;

if (navigator.userAgent.match(botregex)) {

WT.helpers.bdebug.warn('WTO : Bot detected, aborting Optimize setup and data collection.');

WT.optimizeModule.prototype.abort();

return;

}

Bot regex definitions

The below table outlines the automated traffic exclusions included in the example bot regex for bots, crawlers, scrapers, AI agents, search engines, preview fetchers, scanners, and scripted HTTP clients.

User Agent(s)

Classification and purpose

AI2Bot, AI2Bot-DeepResearchEval, Ai2Bot-Dolma, aiHitBot, anthropic-ai, AzureAI-SearchBot, bedrockbot, bigsur.ai, Bravebot, Bytespider, ChatGLM-Spider, ChatGPT Agent, ChatGPT-User, Claude-SearchBot, Claude-User, Claude-Web, ClaudeBot, Cloudflare-AutoRAG, CloudVertexBot, cohere-ai, cohere-training-data-crawler, DeepSeekBot, Gemini-Deep-Research, Google-CloudVertexBot, Google-Extended, Google-Firebase, Google-NotebookLM, GoogleAgent-Mariner, GoogleOther, GoogleOther-Image, GoogleOther-Video, GPTBot, iAskBot, iaskspider, iaskspider/2.0, LinerBot, Manus-User, MistralAI-User, MistralAI-User/1.0, NotebookLM, NovaAct, OAI-SearchBot, OpenAI, Operator, Perplexity-User, PerplexityBot, PhindBot, Poggio-Citations, SBIntuitionsBot, TavilyBot, Thinkbot, TwinAgent, YouBot

AI search, research, training, and browser agents. Included because they may retrieve, analyse, summarise, index, or process pages without representing a normal human visitor viewing an experiment or converting.

AddSearchBot, ApifyBot, ApifyWebsiteContentCrawler, Crawl4AI, Crawlspace, Diffbot, ExaBot, FirecrawlAgent, FriendlyCrawler, IbouBot, ImagesiftBot, imageSpider, img2dataset, ISSCyberRiskCrawler, kagi-fetcher, Kangaroo Bot, KunatoCrawler, laion-huggingface-processor, LAIONDownloader, LCC, LinkupBot, MyCentralAIScraperBot, Panscient, panscient.com, Poseidon Research Crawler, QualifiedBot, QuillBot, quillbot.com, ShapBot, TerraCotta, VelenPublicWebCrawler, Webzio-Extended, wpbot, WRTNBot

Scrapers, extractors, and content-collection tools. Included because they retrieve or extract website content programmatically and may trigger page code without being genuine experiment participants.

amazon-kendra, Amazonbot, AmazonBuyForMe, Amzn-SearchBot, Amzn-User, Andibot, Anomura, atlassian-bot, Awario, Channel3Bot, Cotoyogi, Datenbank Crawler, DuckAssistBot, Echobot Bot, EchoboxBot, ICC-Crawler, Linguee Bot, netEstate Imprint Crawler, PanguBot, PetalBot, Sidetrade indexer bot, Spider, Timpibot, WARDBot, YaK, ZanistaBot

General-purpose, enterprise, product, monitoring, and indexing crawlers. Included because they automatically discover, index, monitor, or inspect content rather than behave like normal human visitors.

Applebot, Applebot-Extended, CCBot, Webspider, webcrawler, Crawler Bot, crawler4j, Crawler/\\d, Google\\s?bot, Mediapartners-Google, AdsBot, Baiduspider, bingbot, YandexAdditional, YandexAdditionalBot, YandexBot, YandexImages, yandex.com/bots, Yahoo! Slurp

Search-engine crawlers and generic spiders. Included because they crawl pages for search indexes, image indexes, advertising systems, or automated discovery.

Google Search Console, Google-Adwords, Google Page Speed Insights, GoogleEarth, GoogleToolbar, Google-Apps-Script, Google Keyword Suggestion, Google-SearchByImage, Google Favicon, Google Web Preview, Google-Site-Verification, google.com/feedfetcher, BingPreview, Yahoo Link Preview, Yahoo Ad monitoring, YahooCacheSystem

Search, preview, verification, advertising, and service fetchers. Included because they request pages or page metadata for previews, validation, feed retrieval, performance checks, advertising, or search features.

AhrefsBot, abevalbot, AlphaBot, deepcrawl, DomainSONOCrawler, GetIntent Crawler, GrapeshotCrawler, klaviyofootprintscanner, lumar, MJ12bot, NetcraftSurveyAgent, NetSeer crawler, onetrustbot, proximic, radius compliance bot, Screaming Frog SEO, seekportbot, SemrushBot, SemrushBot-OCOB, SemrushBot-SWA, sitescorebot, similarweb, SMTBot, TurnitinBot, Wappalyzer, letsencrypt.org

SEO, analytics, auditing, compliance, reputation, and security scanners. Included because they analyse or scan the site automatically and should not consume experiment sessions or influence experiment results.

FacebookBot, facebookexternalhit, FeedBurner/, FeedDemon, MagpieRSS, omgili, omgilibot, rssowl, SimplePie, tt-rss, Twitterbot, WhatsApp/\\d, Youtube-Links, longurl, YOURLS

Social-media previews, RSS readers, feed processors, and URL services. Included because they fetch pages, metadata, feeds, or link previews without representing a person actively browsing the experiment.

KlaviyoAIBot, Mail.RU_Bot, TikTokSpider, GrapeshotCrawler, AmazonBuyForMe, Amzn-User, FacebookBot, Twitterbot, WhatsApp/\\d

Marketing, commerce, social, and messaging automation. Included because these services may inspect pages for product discovery, advertising, link previews, audience analysis, or content sharing.

Apache-HttpClient, AppEngine-Google, Clickagy Intelligence Bot, Cliqzbot, Download Master, Go-http-client, GnuTLS, headlesschrome, Jersey/, libcurl/, okhttp, Photon/\\d, python-requests, riddler.io/, rogerbot, Scrapy, skype-vhod, zgrab, ZmEu

Generic HTTP clients, headless browsers, scraping frameworks, and scanners. Included because they commonly indicate scripted or automated requests. These have a higher false-positive risk because legitimate applications and integrations may use them.

Brightbot 1.0, BuddyBot, Factset_spyderbot, Grammarly, hanaleibot, Meta-ExternalAgent, Meta-ExternalFetcher, meta-webindexer, MaxPointCrawler/, LightspeedSystemsCrawler, LinkpadBot, NetSeer crawler, Storebot, GetIntent Crawler, Klarnabot, pricewatcher, taboolabot, storebot, getthit.com, cocolyzebot, sabot, pingbot, leavemealonebot, pingdom.com, RuxitSynthetic

Other named service, monitoring, commerce, advertising, and utility bots. Included because the names identify automated agents that may monitor, test, compare prices, analyse, or fetch pages without representing a normal experiment visitor.

FeedBurner/, Photon/\\d, Yeti/, Jersey/, bidswitchbot/, riddler.io/, Crawler/\\d, WhatsApp/\\d, Yeti/, YandexImages, Yeti/

Versioned or formatted bot identifiers. Included because the regex matches a family of user agents using a slash or numeric suffix, such as a crawler version or service version.

+https?:\\/\\/[\\w\\.\\-\\/]+(robot|bot|crawler|embed|fetcher|seznambot|feedparser|daum.net|brandwatch|spider|checksite|google.com|goo.gl\\/)

Broad automated-client pattern. Included to catch unknown or newly introduced bots whose user agent contains a URL followed by terms such as bot, crawler, spider, fetcher, or feedparser. This is the broadest pattern and has the highest potential for false positives.

ES6 Check / Abort

This script aborts Optimize if the browser is not ES6 compatible.

var supportsES6 = (function () {

try {

new Function("async () => {}");

new Function("(a = 0) => a");

return true;

}

catch (err) {

return false;

}

}());

if (!supportsES6) {

WT.optimizeModule.prototype.abort();

return;

}

Did this answer your question?