There are many reasons where you could choose for Optimize to not run. This document covers how, where, and a few examples.
How and where
These are all Javascript conditions. We recommend making sure the information is available before the wt.js tag runs on the page, so that we're not waiting for the page to load prior to aborting the tag.
If this is possible, the code you'll write should go into the pre-init/global tag script section of the tag confguration.
If this is not possible, the code should go into the post-load/tag trigger script section, where you can poll for some conditions before running
WT.optimize.setup, and selectively abort at that point. Note though, that polling for conditions slows down page load times.
Notes:
Aborting the tag does not force the rest of the code to stop. You may wish to either use
throw,returnorifconditions to make sure future parts of the tag do not run (if needed).
Examples
Abort the tag for known bots
You may not want known bots to run Optimize - aborting the tag for these will save some of your session usage.
var botregex = /(AddSearchBot|AI2Bot|AI2Bot-DeepResearchEval|Ai2Bot-Dolma|aiHitBot|amazon-kendra|Amazonbot|AmazonBuyForMe|Amzn-SearchBot|Amzn-User|Andibot|Anomura|anthropic-ai|ApifyBot|ApifyWebsiteContentCrawler|Applebot|Applebot-Extended|atlassian-bot|Awario|AzureAI-SearchBot|bedrockbot|bigsur.ai|Bravebot|Brightbot 1.0|BuddyBot|Bytespider|CCBot|Channel3Bot|ChatGLM-Spider|ChatGPT Agent|ChatGPT-User|Claude-SearchBot|Claude-User|Claude-Web|ClaudeBot|lightpanda|Cloudflare-AutoRAG|CloudVertexBot|cohere-ai|cohere-training-data-crawler|Cotoyogi|Crawl4AI|Crawlspace|Datenbank Crawler|DeepSeekBot|Devin|Diffbot|DuckAssistBot|Echobot Bot|EchoboxBot|ExaBot|FacebookBot|facebookexternalhit|Factset_spyderbot|FirecrawlAgent|FriendlyCrawler|Gemini-Deep-Research|Google-CloudVertexBot|Google-Extended|Google-Firebase|Google-NotebookLM|GoogleAgent-Mariner|GoogleOther|GoogleOther-Image|GoogleOther-Video|GPTBot|iAskBot|iaskspider|iaskspider\/2.0|IbouBot|ICC-Crawler|ImagesiftBot|imageSpider|img2dataset|ISSCyberRiskCrawler|kagi-fetcher|Kangaroo Bot|KlaviyoAIBot|KunatoCrawler|laion-huggingface-processor|LAIONDownloader|LCC|LinerBot|Linguee Bot|LinkupBot|Manus-User|Meta-ExternalAgent|Meta-ExternalFetcher|meta-webindexer|MistralAI-User|MistralAI-User\/1.0|MyCentralAIScraperBot|netEstate Imprint Crawler|NotebookLM|NovaAct|OAI-SearchBot|omgili|omgilibot|OpenAI|Operator|PanguBot|Panscient|panscient.com|Perplexity-User|PerplexityBot|PetalBot|PhindBot|Poggio-Citations|Poseidon Research Crawler|QualifiedBot|QuillBot|quillbot.com|RuxitSynthetic|SBIntuitionsBot|Scrapy|SemrushBot-OCOB|SemrushBot-SWA|ShapBot|Sidetrade indexer bot|Spider|TavilyBot|TerraCotta|Thinkbot|TikTokSpider|Timpibot|TwinAgent|VelenPublicWebCrawler|WARDBot|Webzio-Extended|wpbot|WRTNBot|YaK|YandexAdditional|YandexAdditionalBot|YouBot|ZanistaBot|Yahoo Link Preview|Yahoo! Slurp|YOURLS|webcrawler|Crawler Bot|Youtube-Links|Google Search Console|Google-Adwords|Google Page Speed Insights|GoogleEarth|GoogleToolbar|Google-Apps-Script|Google Keyword Suggestion|Google-SearchByImage|Yahoo Ad monitoring|rssowl|yandex.com\/bots|GetIntent Crawler|grammarly|feedly|web spider|Go-http-client|Mediapartners-Google|AdsBot|python-requests|Google\s?bot|AhrefsBot|abevalbot|bingbot|YandexBot|DotBot|sitescorebot|deepcrawl|lumar|similarweb|cocolyzebot|SMTBot|sabot|pingbot|leavemealonebot|pingdom.com|onetrustbot|storebot|getthit.com|taboolabot|radius compliance bot|klarnabot|pricewatcher|moatbot|Baiduspider|GrapeshotCrawler|AlphaBot|proximic|coccocbot|NetcraftSurveyAgent|SemrushBot|seekportbot|Mail.RU_Bot|YandexImages|ZmEu|BingPreview|FeedDemon|Twitterbot|SimplePie|Webspider|CloudFlare-AlwaysOnline|MagpieRSS|longurl|tt-rss|Google Favicon|headlesschrome|GroupHigh|hanaleibot|ia_archiver|google.com\/feedfetcher|MJ12bot|archive.org_bot|Yeti\/|zgrab|vkShare|Screaming Frog SEO|klaviyofootprintscanner|NetSeer crawler|crawler4j|letsencrypt.org|Google-Site-Verification|DomainSONOCrawler|Download Master|WhatsApp\/\d|GnuTLS|Cliqzbot|riddler.io\/|okhttp|filterdb.iss.net\/crawler|Google Web Preview|AppEngine-Google|Jersey\/|DuckDuckGo-Favicons-Bot|rogerbot|YahooCacheSystem|Apache-HttpClient|bidswitchbot\/|Clickagy Intelligence Bot|MaxPointCrawler\/|LightspeedSystemsCrawler|Photon\/\d|FeedBurner\/|Wappalyzer|LinkpadBot|Crawler\/\d|BrokenLinkCheck.com|libcurl\/|TurnitinBot|Microsoft Office Protocol Discovery|Microsoft Windows Network Diagnostics|skype-vhod|\+https?:\/\/[\w\.\-\/]+(robot|bot|crawler|embed|fetcher|seznambot|feedparser|daum.net|brandwatch|spider|checksite|google.com|goo.gl\/))/i;
if (navigator.userAgent.match(botregex)) {
WT.helpers.bdebug.warn('WTO : Bot detected, aborting Optimize setup and data collection.');
WT.optimizeModule.prototype.abort();
return;
}
Bot regex definitions
Bot regex definitions
The below table outlines the automated traffic exclusions included in the example bot regex for bots, crawlers, scrapers, AI agents, search engines, preview fetchers, scanners, and scripted HTTP clients.
User Agent(s) | Classification and purpose |
AI2Bot, AI2Bot-DeepResearchEval, Ai2Bot-Dolma, aiHitBot, anthropic-ai, AzureAI-SearchBot, bedrockbot, bigsur.ai, Bravebot, Bytespider, ChatGLM-Spider, ChatGPT Agent, ChatGPT-User, Claude-SearchBot, Claude-User, Claude-Web, ClaudeBot, Cloudflare-AutoRAG, CloudVertexBot, cohere-ai, cohere-training-data-crawler, DeepSeekBot, Gemini-Deep-Research, Google-CloudVertexBot, Google-Extended, Google-Firebase, Google-NotebookLM, GoogleAgent-Mariner, GoogleOther, GoogleOther-Image, GoogleOther-Video, GPTBot, iAskBot, iaskspider, iaskspider/2.0, LinerBot, Manus-User, MistralAI-User, MistralAI-User/1.0, NotebookLM, NovaAct, OAI-SearchBot, OpenAI, Operator, Perplexity-User, PerplexityBot, PhindBot, Poggio-Citations, SBIntuitionsBot, TavilyBot, Thinkbot, TwinAgent, YouBot | AI search, research, training, and browser agents. Included because they may retrieve, analyse, summarise, index, or process pages without representing a normal human visitor viewing an experiment or converting. |
AddSearchBot, ApifyBot, ApifyWebsiteContentCrawler, Crawl4AI, Crawlspace, Diffbot, ExaBot, FirecrawlAgent, FriendlyCrawler, IbouBot, ImagesiftBot, imageSpider, img2dataset, ISSCyberRiskCrawler, kagi-fetcher, Kangaroo Bot, KunatoCrawler, laion-huggingface-processor, LAIONDownloader, LCC, LinkupBot, MyCentralAIScraperBot, Panscient, panscient.com, Poseidon Research Crawler, QualifiedBot, QuillBot, quillbot.com, ShapBot, TerraCotta, VelenPublicWebCrawler, Webzio-Extended, wpbot, WRTNBot | Scrapers, extractors, and content-collection tools. Included because they retrieve or extract website content programmatically and may trigger page code without being genuine experiment participants. |
amazon-kendra, Amazonbot, AmazonBuyForMe, Amzn-SearchBot, Amzn-User, Andibot, Anomura, atlassian-bot, Awario, Channel3Bot, Cotoyogi, Datenbank Crawler, DuckAssistBot, Echobot Bot, EchoboxBot, ICC-Crawler, Linguee Bot, netEstate Imprint Crawler, PanguBot, PetalBot, Sidetrade indexer bot, Spider, Timpibot, WARDBot, YaK, ZanistaBot | General-purpose, enterprise, product, monitoring, and indexing crawlers. Included because they automatically discover, index, monitor, or inspect content rather than behave like normal human visitors. |
Applebot, Applebot-Extended, CCBot, Webspider, webcrawler, Crawler Bot, crawler4j, Crawler/\\d, Google\\s?bot, Mediapartners-Google, AdsBot, Baiduspider, bingbot, YandexAdditional, YandexAdditionalBot, YandexBot, YandexImages, yandex.com/bots, Yahoo! Slurp | Search-engine crawlers and generic spiders. Included because they crawl pages for search indexes, image indexes, advertising systems, or automated discovery. |
Google Search Console, Google-Adwords, Google Page Speed Insights, GoogleEarth, GoogleToolbar, Google-Apps-Script, Google Keyword Suggestion, Google-SearchByImage, Google Favicon, Google Web Preview, Google-Site-Verification, google.com/feedfetcher, BingPreview, Yahoo Link Preview, Yahoo Ad monitoring, YahooCacheSystem | Search, preview, verification, advertising, and service fetchers. Included because they request pages or page metadata for previews, validation, feed retrieval, performance checks, advertising, or search features. |
AhrefsBot, abevalbot, AlphaBot, deepcrawl, DomainSONOCrawler, GetIntent Crawler, GrapeshotCrawler, klaviyofootprintscanner, lumar, MJ12bot, NetcraftSurveyAgent, NetSeer crawler, onetrustbot, proximic, radius compliance bot, Screaming Frog SEO, seekportbot, SemrushBot, SemrushBot-OCOB, SemrushBot-SWA, sitescorebot, similarweb, SMTBot, TurnitinBot, Wappalyzer, letsencrypt.org | SEO, analytics, auditing, compliance, reputation, and security scanners. Included because they analyse or scan the site automatically and should not consume experiment sessions or influence experiment results. |
FacebookBot, facebookexternalhit, FeedBurner/, FeedDemon, MagpieRSS, omgili, omgilibot, rssowl, SimplePie, tt-rss, Twitterbot, WhatsApp/\\d, Youtube-Links, longurl, YOURLS | Social-media previews, RSS readers, feed processors, and URL services. Included because they fetch pages, metadata, feeds, or link previews without representing a person actively browsing the experiment. |
KlaviyoAIBot, Mail.RU_Bot, TikTokSpider, GrapeshotCrawler, AmazonBuyForMe, Amzn-User, FacebookBot, Twitterbot, WhatsApp/\\d | Marketing, commerce, social, and messaging automation. Included because these services may inspect pages for product discovery, advertising, link previews, audience analysis, or content sharing. |
Apache-HttpClient, AppEngine-Google, Clickagy Intelligence Bot, Cliqzbot, Download Master, Go-http-client, GnuTLS, headlesschrome, Jersey/, libcurl/, okhttp, Photon/\\d, python-requests, riddler.io/, rogerbot, Scrapy, skype-vhod, zgrab, ZmEu | Generic HTTP clients, headless browsers, scraping frameworks, and scanners. Included because they commonly indicate scripted or automated requests. These have a higher false-positive risk because legitimate applications and integrations may use them. |
Brightbot 1.0, BuddyBot, Factset_spyderbot, Grammarly, hanaleibot, Meta-ExternalAgent, Meta-ExternalFetcher, meta-webindexer, MaxPointCrawler/, LightspeedSystemsCrawler, LinkpadBot, NetSeer crawler, Storebot, GetIntent Crawler, Klarnabot, pricewatcher, taboolabot, storebot, getthit.com, cocolyzebot, sabot, pingbot, leavemealonebot, pingdom.com, RuxitSynthetic | Other named service, monitoring, commerce, advertising, and utility bots. Included because the names identify automated agents that may monitor, test, compare prices, analyse, or fetch pages without representing a normal experiment visitor. |
FeedBurner/, Photon/\\d, Yeti/, Jersey/, bidswitchbot/, riddler.io/, Crawler/\\d, WhatsApp/\\d, Yeti/, YandexImages, Yeti/ | Versioned or formatted bot identifiers. Included because the regex matches a family of user agents using a slash or numeric suffix, such as a crawler version or service version. |
+https?:\\/\\/[\\w\\.\\-\\/]+(robot|bot|crawler|embed|fetcher|seznambot|feedparser|daum.net|brandwatch|spider|checksite|google.com|goo.gl\\/) | Broad automated-client pattern. Included to catch unknown or newly introduced bots whose user agent contains a URL followed by terms such as bot, crawler, spider, fetcher, or feedparser. This is the broadest pattern and has the highest potential for false positives. |
ES6 Check / Abort
This script aborts Optimize if the browser is not ES6 compatible.
var supportsES6 = (function () {
try {
new Function("async () => {}");
new Function("(a = 0) => a");
return true;
}
catch (err) {
return false;
}
}());
if (!supportsES6) {
WT.optimizeModule.prototype.abort();
return;
}
