# Can Bots See Me reference The deep material behind https://canbotsseeme.pages.dev/agent.md: engine-by-engine facts, bots, studies, content evidence, Search Console, measuring AI answers and every check id. Start with https://canbotsseeme.pages.dev/agent.md; fetch this only when a task needs the detail. Method and every check: https://canbotsseeme.pages.dev/method (JSON: https://canbotsseeme.pages.dev/api/method). What was cut: https://canbotsseeme.pages.dev/changelog. ## Evidence - Evidence: every claim here carries its grade (Official = the vendor's own docs, Observed = measured or reported by someone other than the engine's vendor (often a commercial study; conflicts noted where known), Speculative = inference) and source keys (resolved at https://canbotsseeme.pages.dev/api/citations); every check has https://canbotsseeme.pages.dev/method#. Do not present Observed or Speculative items as rules. - The readiness score is not a visibility forecast: in a within-site study of API answers (small models: ChatGPT's API with GPT-5.4 nano, Gemini 3.1 Flash Lite, Perplexity Sonar; 651 questions written for one US fiber provider, plus a niche B2B set; a preprint; one author advises a GEO vendor that helped with the data, and the questions were written for the authors' monitoring-platform clients), page features explained 2.8% of citation differences between pages of one site for consumer queries against about 32% for the retrieval of ChatGPT's API, measured by replaying the questions through OpenAI's API (in a smaller niche B2B set 10% against 30%, not a significant difference); a query-blind score tracked citation order at ρ ≈ 0.11 in a fixed-source lab test by the score's vendor; and in a single-author thesis study of 112 startups queried through two model APIs, an on-page GEO score did not correlate with discovery (Observed · 08-verification #41, 08-verification #42, 04-academic #20). Claim a traffic effect only when it was measured against untouched pages. ## 1. Set up - Get the site URL. Check whether you are inside its codebase (package.json, next.config.*, astro.config.*, nuxt.config.*, wp-content, hugo.toml, plain HTML); if not, audit and plan, and do provider work through APIs, CLIs or a browser. - Read `.canbotsseeme/context.md` (Business, Goal, Audience, Competitors, Key pages, Writing rules, Research log). If missing: infer the business from the site, confirm it in one question, create the file. Reuse research-log entries under 30 days old. - MCP: `claude mcp add --transport http canbotsseeme https://canbotsseeme.pages.dev/mcp`. Tools: `audit_site` (store: true to get a report id), `get_master_prompt`, `crawl_site`, `check_links`, `fix_recipe`, `keyword_ideas`, `post_briefs`, `send_brief_results`, `research_prompts`. ## 2. Audit and get the plan - GET https://canbotsseeme.pages.dev/api/audit?url=&pages=3 (stored by default, returns `id`; &store=0 skips; &probe=1 tests the CDN with AI bots' user agents, spoofed from Cloudflare IPs, so confirm in logs; &indexnowKey= verifies IndexNow; add &indexnowKeyLocation= when the key file is not /.txt). Report: overall (0–100), stages, findings[] (id, stage, status, detail, why, fix, grade, cites), manual[], pages[]. - GET https://canbotsseeme.pages.dev/api/crawl?url=&max=50 (site-wide, up to ~45 s) and https://canbotsseeme.pages.dev/api/links?url= (one page, external links too). - **Master prompt**: GET https://canbotsseeme.pages.dev/api/report//prompt.md (MCP `get_master_prompt`): the whole run as one plan: every fail/warn with evidence, grade, /method link and fix steps; manual checks as agent tasks with the one owner step; post briefs with the live-results check; verification; a JSON report-back. Pick tasks by the keys in its "### n." list: `?only=k1,k2` (exact set), `?skip=k`, `?add=brief-1` (new-content tasks `brief-` are off unless added), `?run=links,crawl` (accepted, rarely needed: the report's saved crawl and link-check findings are included, and a check that never ran becomes a run-it-yourself task). Following it covers sections 3, 4 and 9. ## 3. Triage - Stages: Access (`eligible`) → Index (`indexed`) → AI answers (`quotable`) → Search basics (`foundations`). An Access fail is a hard blocker: fix it first, nothing else is trustworthy until bots get the real page. - Then fail before warn, Official before Observed before Speculative. Skip status info and myth; checks marked "not scored" below never count. - Lead with one to three changes most likely to win qualified search traffic or stop a real loss. Show the owner a short table: finding, one-line why, the change, whether it needs them. ## 4. Fix - Per finding: GET https://canbotsseeme.pages.dev/api/recipe/ (`bot-` ids share the `bots` recipe; manual ids have recipes too). Every recipe is agent work: make the smallest correct change in the repo or through the provider's API or CLI. An `owner` field is the one step only a person can take (sign-in or API token, payment, business decision, physical action): do the rest and give them that step with its link. - Fix shared templates first. One commit per stage. Verify before deploying: `curl -s ` shows the change in the initial HTML; `curl -sI` for robots.txt, sitemap and headers. ## 5. Bots and engines - Search and answer bots, never block them: OAI-SearchBot, Googlebot, Bingbot, Claude-SearchBot, Claude-User, PerplexityBot, Applebot, Meta-WebIndexer, Amzn-SearchBot, MistralAI-Index, ShapBot (a block on ShapBot warns; the rest fail). - User-triggered and agent fetchers (a person asked an assistant about the page; leave them allowed unless the owner wants out): ChatGPT-User (robots.txt: may not apply), Google-Agent (robots.txt: generally ignored), Google-GeminiNotebook (robots.txt: generally ignored), Perplexity-User (robots.txt: generally ignored), Meta-ExternalFetcher (robots.txt: may bypass), Amzn-User (robots.txt: may not follow all), MistralAI-User (robots.txt: not stated), Shap-User (robots.txt: no). Those that ignore robots.txt can only be stopped at the CDN. - Training bots, the owner's choice: GPTBot, Google-Extended, ClaudeBot, Applebot-Extended, Meta-ExternalAgent, Amazonbot, MistralAI-Training, CCBot. Blocking them keeps search visibility, with three exceptions: Google-Extended also controls grounding in the Gemini app and Vertex AI, Meta-ExternalAgent also indexes, and Amazonbot also improves Amazon products (Content Partners benefits). Google-Extended and Applebot-Extended are robots.txt tokens only. Microsoft has no training token: its page-level opt-outs are noindex or noarchive (no training) and nocache (URL, title and snippet only), and noarchive is Amazon's page-level one. - Ads and preview bots: OAI-AdsBot, AdIdxBot, Meta-ExternalAds, MicrosoftPreview, FacebookExternalHit. OAI-AdsBot and AdIdxBot check ad landing pages (needed when the owner advertises there); Meta-ExternalAds crawls "for use cases such as improving advertising and other business-related products and services" (Meta). (Official · 02-openai #1, 08-verification #33, 03-other-engines #69) - Shopping crawlers: Storebot-Google (Google Shopping surfaces; Googlebot rules do not cover it). Allow it on product pages when the site sells products. (Official · 01-google #5) → `bots-shopping` - robots.txt (RFC 9309, matched like google/robotstxt): 4xx except 429 = crawl everything; more than five redirects = treated as unavailable; 5xx = the RFC says assume complete disallow, Google stops for 12 hours then uses its last good copy for up to 30 days, Amazon treats it as absent; 429 = a server error for Google. Google reads only the first 500 KiB. Changes take up to about 24 hours. Unnamed Applebot follows Googlebot; unnamed Amzn-SearchBot follows the other search bots. (Official · 08-verification #1, 08-verification #2, 03-other-engines #44) → `robots-error`, `robots-missing`, `robots-refused`, `robots-size` - Verify bots by user agent plus the vendor's own proof, never the user agent alone: published IP lists (OpenAI https://openai.com/searchbot.json, https://openai.com/chatgpt-user.json, https://openai.com/gptbot.json, https://openai.com/adsbot.json; Anthropic https://claude.com/crawling/bots.json; Perplexity https://www.perplexity.com/perplexitybot.json and perplexity-user.json; Mistral https://mistral.ai/mistralai-index-ips.json and mistralai-user-ips.json; Amazon https://developer.amazon.com/amazonbot/ip-addresses/ (also searchbot- and live-ip-addresses/); Apple https://search.developer.apple.com/applebot.json; Common Crawl https://index.commoncrawl.org/ccbot.json; Parallel https://docs.parallel.ai/resources/shapbot.json; Google under https://developers.google.com/static/crawling/ipranges/), reverse DNS plus a forward check (Googlebot: googlebot.com, google.com, googleusercontent.com; Bingbot: search.msn.com, or bingbot.json; Applebot: applebot.apple.com; CCBot: crawl.commoncrawl.org, IPv4 only, its IPv6 range is in ccbot.json), or a Web Bot Auth signature (RFC 9421 Signature-Agent and Signature-Input; Cloudflare counts signed agents as verified). Cloudflare: allow verified bots (`cf.client.bot`), but PerplexityBot was removed from that list, so allow Perplexity by user agent plus IP. (Official · 02-openai #1, 03-other-engines #5, 08-verification #5, 08-verification #32, 08-verification #29, 08-verification #28) → `waf-probe`, `waf-allowlist` - Cloudflare (since 2026-09): three AI bot policies, Search, Training and Agent, replace "Block AI bots"; Bot Preference Sync replaces managed robots.txt and writes a "BEGIN Cloudflare Managed" block. Agent = "Block on pages with ads" (the default for new ad-supported sites, and where "Block AI bots" was on) blocks ChatGPT-User, Claude-User and the like there. Opt out of training with "Disallow AI Training", never "Block" or "Block on pages with ads": since 2026-09-15 Block also blocks Googlebot, Bingbot and Applebot, and Block on pages with ads blocks them on every page detected as serving ads. "Disallow AI Training" also blocks Amazonbot, Meta-ExternalAgent and CCBot at the edge (Cloudflare classes them as training crawlers), losing Meta-ExternalAgent's product indexing (Meta AI search runs on Meta-WebIndexer, which Cloudflare classes as a link-preview bot, not a training crawler) and Amazon Content Partners eligibility; to keep those, set Training to Allow and disallow the chosen tokens in robots.txt. Pay per crawl answers crawlers with HTTP 402. (Official · 08-verification #25, 08-verification #26, 08-verification #27, 08-verification #62, 03-other-engines #69, 03-other-engines #44, 08-verification #73) → `waf-allowlist`, `waf-probe` - xAI: a documented absence. Its docs name no crawler or robots.txt token, so do not add "GrokBot" rules; its Grok Bot agent browses from shared static datacenter ranges (not published; xAI gives them to customers through their account team), or through a member's own network, so it can only be managed at the CDN and not reliably by IP. (Official · 08-verification #55) **Google (AI Overviews, AI Mode)** - Same index, ranking and spam systems as Search: the page must be indexed and snippet-eligible (HTTP 200, not blocked, indexable). No extra on-page requirements. (Official · 01-google #2, 01-google #13) → `http-status` - Search Console "Search generative AI control": Google's AI optimization guide (2026-07) says a site must be Included to be eligible; its older AI features page (2025-12) still says "no additional technical requirements". Google's pages conflict, so treat Include as required. Include is the default; Inherit on a child property is fine, Exclude is not. Exclude removes links and grounding in AI Overviews, AI Mode and Discover. No API: read it in the owner's signed-in browser or ask the owner. (Official · 01-google #1, 01-google #38, 01-google #2) → `gsc-genai-control` - robots.txt for Googlebot is the only crawl control for AI Overviews and AI Mode; the display controls are noindex, nosnippet, max-snippet, data-nosnippet and the site-wide generative AI setting. Google-Extended controls Gemini training and grounding in the Gemini app and Vertex AI, not AI Overviews, AI Mode or ranking; it is a robots.txt token only. (Official · 01-google #2, 01-google #5, 08-verification #4) → `bots-training` - nosnippet blocks use as a direct input to AI Overviews and AI Mode; max-snippet caps how much is used (structured data and licensed content excepted); data-nosnippet works only on span, div and section and must not be toggled with JavaScript. Robots meta tags count anywhere in the page, even in . (Official · 01-google #4) → `snippet` - Core Web Vitals "are used by our ranking systems"; measure them with the free PageSpeed Insights API (no key needed). A search and UX fix, not an AI lever (see Not levers). (Official · 08-verification #52, 08-verification #63) → `core-web-vitals` - Googlebot reads only the first 2 MB of an HTML file (uncompressed, headers included) and of each script or stylesheet it renders with; it will not render JavaScript from files robots.txt blocks. (Official · 01-google #146, 08-verification #7, 07-seo-foundations #21) → `size` - Google reads og:title for title links, og:image (with schema.org image) for thumbnails in Search and Discover, og:site_name and WebSite markup for site names, and a home-page for favicons. (Official · 07-seo-foundations #1, 08-verification #12, 08-verification #9, 08-verification #10) → `open-graph` - Query fan-out: AI answers cite many pages that do not rank for the literal query (the share of AI Overview citations from the top 10 fell to about 38%). Cover the sub-questions on real pages; creating one page per query variant primarily to manipulate rankings or AI responses is scaled-content abuse. (Observed · 01-google #125; Official · 01-google #1, 01-google #10) - AI Overview and AI Mode clicks and impressions sit inside Search Console's Web totals and cannot be separated there. The Generative AI reports (Search, and Discover) have impressions only (no queries, no clicks), hourly to monthly, a "Web: multimodal" filter, and no API. (Official · 01-google #41, 01-google #39, 07-seo-foundations #15, 08-verification #49, 08-verification #50) → `gsc-genai-report` - Verify Google's crawlers by IP or reverse DNS: the IP lists moved to https://developers.google.com/static/crawling/ipranges/ (common-crawlers.json for Googlebot, user-triggered-fetchers*.json, user-triggered-agents.json for Google-Agent); reverse DNS ends in googlebot.com, google.com or googleusercontent.com. Google-Agent is testing Web Bot Auth (agent.bot.goog). (Official · 08-verification #5, 08-verification #6, 01-google #6) → `waf-allowlist` - Publishers: sites users mark as Preferred sources get a badge in Top Stories and in AI Overviews/AI Mode; it needs the AI control on Include. (Official · 01-google #16) - Shopping: Merchant Center feeds; its AI performance insights show share of voice and missing attributes for AI shopping queries (English, in AU, CA, IN, NZ and the US). (Official · 01-google #45) → `commerce-feeds` **ChatGPT** - Inclusion in ChatGPT search needs OAI-SearchBot allowed in robots.txt and the host or CDN allowing OpenAI's published searchbot IPs. GPTBot is training only; blocking it does not remove the site from search. (Official · 02-openai #3, 02-openai #1, 02-openai #32) → `bot-` - Sites that opt out of OAI-SearchBot "can still appear as navigational links". OpenAI's retired Publishers FAQ (replaced in October 2026) said only a crawlable noindex suppresses that; no live OpenAI page documents noindex now. (Official · 02-openai #1) → `noindex` - ChatGPT runs its own index: in free instant mode most results came from it, paid Thinking drew most from Google. A good Google rank is not a proxy for ChatGPT: even combined, Google results and OpenAI's Responses API recovered only about a quarter of the URLs ChatGPT Web cited. (Observed · 08-verification #43, 08-verification #41, 08-verification #67) - OpenAI's own index can miss pages that are robots- or CDN-blocked, JavaScript-dependent, behind a login, new, rarely accessed or hard to discover. (Official · 02-openai #6) → `no-js` - ChatGPT's page reader (the fetches behind Thinking-mode opens) runs no JavaScript, strips JSON-LD (keep facts in visible text) and rejects pages over 4 MB; one site owner has logged OAI-SearchBot and GPTBot rendering JavaScript on their sites (several, on one server) since 2026-09-25, so keep core content in the HTML either way. Instant mode mostly runs one search round (Thinking often several; an academic study) and, in one team's measurement, judges a page by its title plus a ~200-character snippet from the start of the body, which usually begins with the H1 (about 150 characters of content after it); the meta description is ignored on OpenAI's own index (used about a third of the time on Google-fed results). No OpenAI page documents any of this. (Observed · 02-openai #80, 08-verification #66, 08-verification #67) → `quick-mode` - OpenAI's reading cache was observed keeping pages that carry noindex; noindex is not a way out of it. (Observed · 08-verification #43) - A title that semantically matches the fan-out queries predicts citation; matching the user's literal question went with no more citations in ChatGPT's API answers (−0.2%) in a within-site study (an association). (Observed · 02-openai #88, 08-verification #41) → `title` - Most ChatGPT clicks carry no referrer, so utm_source=chatgpt.com is the signal (OpenAI's page that documented it was retired). Thinking-mode opens can lack it, and links lost it from 2026-10-08 to 10-15 (a bug OpenAI confirmed): annotate that week as a tracking gap. (Observed · 02-openai #80, 08-verification #64) → `analytics` - OpenAI's always-on agents (dots) browse from their own cloud browser, which OpenAI documents (some sites block it) without a user agent or IP list, so rules and log reports keyed to its four published bots miss them. ChatGPT Work's Cloud browser does sign its requests with Web Bot Auth (Signature-Agent "https://chatgpt.com"; Cloudflare tag chatgpt-agent); whether dots use it is not documented. (Official · 08-verification #70, 08-verification #71; Observed · 08-verification #65) → `server-logs` - After a 7 May 2026 change, about 60% of ChatGPT referral clicks landed on home pages (up from 26–32%; a vendor's desktop panel over three weeks): the home page must work for a cold, high-intent visitor. (Observed · 02-openai #108) - Shopping (US only): Shopify and Etsy catalogs are already integrated; other merchants apply to OpenAI's product feed program (waitlist, daily snapshot in OpenAI's schema); a feed is optional when ChatGPT already crawls the site; accepted feeds are delivered by SFTP push, OpenAI's feed API or a supported commerce platform or feed provider, not fetched from a URL; OAI-SearchBot must reach product image URLs. Merchant rank uses availability, price, quality and maker or primary-seller status. (Official · 02-openai #4, 08-verification #38, 08-verification #39, 08-verification #69, 02-openai #8) → `commerce-feeds` - Verify OpenAI's bots with its IP lists (https://openai.com/searchbot.json, chatgpt-user.json, gptbot.json, adsbot.json). OpenAI suggests allowlisting by its user agents; pair them with the IP lists, since user agents can be spoofed. robots.txt changes take about 24 hours. (Official · 02-openai #1, 02-openai #8) → `waf-allowlist` **Copilot and Bing** - The Bing index grounds Copilot and Bing AI summaries. Microsoft also says "Microsoft grounding powers nearly every major AI assistant" (its claim about its grounding service, not a measured reach). (Official · 03-other-engines #10, 03-other-engines #6) → `bot-` - NOARCHIVE excludes a page from Copilot answers; NOCACHE limits it to URL, title and snippet; both together count as NOCACHE. Microsoft's page-level training opt-outs are noindex or noarchive (no training) and nocache (URL, title and snippet only); there is no robots.txt token. nosnippet hides captions and "may limit Copilot citation quality". data-nosnippet hides a passage (any element) and keeps the page ranking, but Bing says it too "may limit Copilot citation quality" (Bing does not say whether only for the wrapped text). (Official · 03-other-engines #6, 08-verification #31, 03-other-engines #14) → `copilot-directives` - IndexNow for changes (deleted and redirected URLs too); sitemaps with only canonical, live URLs and an accurate lastmod: freshness signals "directly influence" how fast updates reach AI answers. ETags help Bing detect changes. Use 302 only for moves shorter than two days. (Official · 03-other-engines #6, 03-other-engines #16) → `indexnow` - Near-duplicate pages are clustered and one representative is picked, possibly an outdated one: consolidate duplicates (301 or canonical). (Official · 03-other-engines #15) → `site-duplicates` - Bing's grounding index may abstain when support is missing, stale or conflicting. (Official · 03-other-engines #11) - Bing Webmaster Tools AI Performance: citations, cited pages, sampled grounding queries, intents, Citation Share, Compare and Topics Portfolio across Copilot, Bing AI summaries and select partners; CSV export. Free. (Official · 03-other-engines #7, 03-other-engines #8, 08-verification #35, 08-verification #36) → `bing-ai-performance` - Verify Bingbot by reverse DNS (search.msn.com plus a forward check) or the bingbot.json list (refresh daily). AdIdxBot checks Microsoft Advertising landing pages; MicrosoftPreview renders previews. (Official · 08-verification #32, 08-verification #33) → `waf-allowlist` - Microsoft Merchant Center feeds inform organic Copilot product results; PayPal Store Sync and Shopify make catalogs purchasable in Copilot Checkout. (Official · 03-other-engines #37, 03-other-engines #39) → `commerce-feeds` **Claude** - Anthropic lists Brave Search, TurboPuffer and Parallel Web Systems (since 2026-10-14) as web-search subprocessors, and Claude-SearchBot builds an index of its own. Brave is one confirmed provider, not the only one. One summer-2026 study of Claude's API web-search tool, run by an AI-SEO agency (CompleteSEO) on its own clients' queries (1,481 matched queries; conflict of interest: it sells AI SEO services), found 87.8% of retrieved URLs in Brave's top 10, against 45.6% for Google, before Parallel was added. (Official · 08-verification #22, 08-verification #23, 03-other-engines #5; Observed · 03-other-engines #25) → `brave-ranking` - Brave's crawler has no user agent of its own and won't crawl what Googlebot can't; robots.txt alone does not delist a page from Brave; a robots noindex does (and a 404 or 410 page drops out once Brave re-fetches it). search.brave.com/submit-url is a form to have a URL "re-fetched" (Brave lists it for an update, a noindex or a removal); a page Brave has not indexed yet has no documented submission route (for a page that no longer exists, Brave also lists not-found@brave.com). Brave discovers pages by crawling (only what Googlebot may crawl) and partly from opt-in Web Discovery Project data, so keep key pages crawlable and linked (inferred). (Official · 03-other-engines #87, 08-verification #72) → `bot-` - ShapBot builds Parallel's index and obeys robots.txt; Shap-User (user fetches) is not managed by robots.txt. Claude's image results come from Bing. (Official · 08-verification #24, 08-verification #40) - ClaudeBot is training; Claude-SearchBot and Claude-User honor robots.txt and will not bypass CAPTCHAs. Claude's web fetch does not render JavaScript. One IP list covers all three bots: https://claude.com/crawling/bots.json; Anthropic says IP blocking is not a reliable opt-out (use robots.txt). Anthropic documents robots.txt only, not noindex. (Official · 03-other-engines #5, 03-other-engines #92) → `no-js` **Perplexity** - Its own index (PerplexityBot, not training). Stale results are filtered out before ranking, and pages are ranked as whole documents and as passages. (Official · 03-other-engines #28) - Perplexity-User generally ignores robots.txt. robots.txt changes take up to 24 hours. IP lists: https://www.perplexity.com/perplexitybot.json and https://www.perplexity.com/perplexity-user.json. Perplexity documents robots.txt only, not noindex. (Official · 03-other-engines #1) → `bots-user` - Cloudflare saw Perplexity fetch with an undeclared browser user agent when PerplexityBot was blocked, and removed PerplexityBot from its verified-bot list in 2025: allow it by user agent plus IP, not cf.client.bot. (Observed · 03-other-engines #3; Official · 08-verification #28) - Shopping: the free Merchant Program, aimed at large retailers, raises the "chances of being a recommended product". (Official · 03-other-engines #30) → `commerce-feeds` **Apple, Meta, Amazon, Mistral, Common Crawl** - Applebot powers Siri, Spotlight and Safari and gives context for Apple's AI answers; nosnippet opts a page out of that answer use, and so does isAccessibleForFree: false (hasPart is not supported, so Google's metered-paywall pattern takes the whole page out). If robots.txt names Googlebot but not Applebot, Applebot follows the Googlebot rules. Applebot-Extended controls training only and sends no requests. (Official · 03-other-engines #43) → `apple-paywall` - Meta-WebIndexer feeds Meta AI search ("helps us cite and link to your content"); Meta-ExternalAgent is training and indexing; Meta-ExternalFetcher may bypass robots.txt. Meta documents robots.txt, not noindex. (Official · 03-other-engines #69) - Amzn-SearchBot feeds Alexa search (no training) and, when not named, follows the rules set for other search bots. Amazonbot is not training-only: it improves Amazon products and allowing it is tied to Amazon Content Partners benefits. Amazon honours page-level noindex, none and noarchive ("do not use the page for model training"), and receives IndexNow. (Official · 03-other-engines #44, 08-verification #34) → `bots-training` - MistralAI-Index is Mistral's search index for Vibe (not training); Mistral lists MistralAI-User as a robots.txt token without saying it obeys robots.txt. (Official · 03-other-engines #45) - CCBot (Common Crawl) checks robots.txt first and obeys Crawl-delay; its archive is widely used for training. Verify by https://index.commoncrawl.org/ccbot.json, or for IPv4 by reverse DNS (crawl.commoncrawl.org); its IPv6 range has no reverse DNS. (Official · 08-verification #68, 08-verification #21) ## 6. Content **Levers** (what to change): - Substance: original data, first-hand testing and detail, expert judgement, specific cases. Google calls non-commodity content the thing that matters most and says AI features need no special optimization. Check what the top results lack live (Briefs below); never guess. (Official · 01-google #1, 01-google #2, 01-google #7) - Get retrieved by each engine's own search: in a within-site study of API answers (small models: ChatGPT's API with GPT-5.4 nano, Gemini 3.1 Flash Lite, Perplexity Sonar; 651 questions written for one US fiber provider, plus a niche B2B set; a preprint; one author advises a GEO vendor that helped with the data, and the questions were written for the authors' monitoring-platform clients), the retrieval of ChatGPT's API, measured by replaying the questions through OpenAI's API, explained about 32% of which pages got cited, against 2.8% for page features (consumer queries; in a smaller niche B2B set 10% against 30%, not a significant difference). Engines shared almost no cited URLs in a small one-day audit (author at an AI-visibility firm), and one engine's cited pages change day to day and drift over weeks, so measure per engine. Cover the fan-out (the sub-questions, comparisons, constraints, pricing and specs a searcher or an AI planner asks next) on real pages with depth. (Observed · 08-verification #41, 08-verification #60, 01-google #125, 05-industry-studies #13; Official · 01-google #1) - A body paragraph that answers the query in its own terms: in the same study the best-matching paragraph went with more citations in every engine's API answers (+15.5% per SD pooled, wherever it sat; weaker on brands' own pages, +5.2%), and a second within-domain study (agency authors) agrees. Associations, not experiments. (Observed · 08-verification #41, 08-verification #59) → `gsc-striking` - Title: the page's topic and main search phrase first, brand last, unique per page. Use the query's own wording where it reads naturally (in API answers it went with more citations on Gemini and Perplexity, not ChatGPT, which rewrites the question). (Official · 07-seo-foundations #1; Observed · 02-openai #88, 08-verification #41) → `title` - A unique meta description (name="description") that says in plain words what the page offers. (Official · 07-seo-foundations #18) → `meta-desc` - Main content and core facts in the initial HTML, readable without JavaScript, as text rather than only in images or PDFs. (Official · 03-other-engines #6, 03-other-engines #92, 07-seo-foundations #21; Observed · 02-openai #80) → `no-js` - Specific, consistent facts (prices, locations, founders, specs) stated on the site's own pages: for factual details retrieved evidence usually overrides a model's prior, and vague official pages lose to specific third-party claims. For which brand gets recommended, the model's prior dominates for well-known brands; niche brands depend on retrieved evidence. (Observed · 04-academic #83, 05-industry-studies #116, 08-verification #41) - Authorship you can stand behind: real bylines, author pages, how the content was made. Never fabricate personas. (Official · 01-google #7) - Genuine updates: change the content, then show "Updated " and dateModified. Google lists changing a page's date without substantially changing it as a sign of search-engine-first content. Cited pages skew fresher, most on ChatGPT, and in controlled tests a recent date helps once a page is retrieved; the live effect is small and unproven (not scored here). (Official · 01-google #7; Observed · 05-industry-studies #76, 04-academic #7, 08-verification #44) → `freshness` - Images and video that carry real information (Google recommends multimodal assets for AI features). (Official · 01-google #1, 01-google #3) - Internal links: plain links with descriptive, varied anchors; every important page linked from another page (see Internal linking). (Official · 07-seo-foundations #4) → `internal-links` - Third-party presence: being named on the pages engines cite (reviews, comparisons, earned media, YouTube, forums). Controlled tests show a model's familiarity with a brand reorders its recommendations, and niche brands are named mainly when a cited third-party page names them; whether building mentions changes that prior is untested. The agent maps which sources get cited (research prompts); outreach is the owner's, and inauthentic mention-seeking does not help. (Observed · 08-verification #41, 05-industry-studies #16, 05-industry-studies #24; Official · 01-google #1) - Links from other sites to a specific page went with more citations of that page within a site in API answers (most on Gemini's API; direction not established), while domain authority added little. Earn links (PR, genuinely useful resources); never buy them, which is link spam. (Observed · 08-verification #41; Official · 01-google #10) **Hygiene, not levers:** - Structured data: valid JSON-LD that matches the visible content, meets each rich-result feature's rules, and Organization (or Person) on the home or About page for entity clarity. Causal and within-site studies found no citation lift from adding schema, and Google requires no special schema for AI features. (Official · 01-google #21, 07-seo-foundations #12, 01-google #1; Observed · 05-industry-studies #59, 08-verification #41) → `rich-results` - Crawl hygiene: CSS, JavaScript and preview images crawlable; links as ; 301 for permanent moves; a favicon and WebSite name on the home page; ETag or Last-Modified; the same robots meta, structured data and content on mobile and desktop. (Official · 07-seo-foundations #21, 07-seo-foundations #4, 08-verification #9, 08-verification #8, 07-seo-foundations #6) → `resources-blocked` - Never: hidden text, instructions aimed at AI assistants (treated as manipulation), cloaking, bought links, or a near-duplicate page per city or keyword variant. (Official · 01-google #10, 02-openai #31, 03-other-engines #6) → `injection` **Not levers** (reported at most, never rules; https://canbotsseeme.pages.dev/changelog): - llms.txt: 97% of files got zero requests in a 137K-site study, and of the 3% that were fetched, 77% of the bots reading them were not AI tools: AI bots made 19.5% of fetches, GPTBot (a training crawler) was the top AI fetcher, and coding agents such as Claude-Code out-fetched every AI retrieval bot (Ahrefs, a vendor; requests, not use); no study found an effect on citations. Google Search ignores it ("neither harm nor help"). The proposal (v2, 2026) targets agents at inference, and Chrome Lighthouse audits it for server errors. Reported as a myth, never scored. (Observed · 05-industry-studies #64, 03-other-engines #54; Official · 01-google #1, 08-verification #17, 08-verification #16) → `llms-txt` - Schema as a ranking or citation lever: a difference-in-differences test on 1,885 pages already heavily cited by AI found none (AI Overviews −4.6%, others noise), and a within-site study found none either. Keep valid markup as hygiene only. (Observed · 05-industry-studies #59, 08-verification #41; Official · 01-google #1) → `schema` - Answer-first openings, "key information early" (secondary at most): Bing's guidelines ask for them and Google says no special optimization. Pooled over API answers, a matching first paragraph went with smaller gains (+5.9% per SD, mainly Gemini's API) than the best-matching paragraph (+15.5%); on brands' own pages the first paragraph did slightly better (+7.8% vs +5.2%), and in a niche B2B set not at all. Two studies put the best paragraph a median 36–45% down the page. Not scored: write a clear opening that answers the question, never restructure pages for bots. (Official · 03-other-engines #6, 01-google #2; Observed · 08-verification #41, 08-verification #59) - Question-shaped headings, FAQ blocks and heading order: FAQ blocks show no within-site effect, and Google says heading order does not matter. Question-shaped headings were not tested within a site. The heading count was mixed (−2.7% consumer, +7.7% niche B2B), and one cross-site vendor study on ChatGPT found pages whose headings closely match the query (by embedding similarity) cited more often (41% vs 30%). Use a heading that names the topic; what goes with more citations is a title and a paragraph that answer the question in its own terms. Don't rewrite headings as questions for bots. (Observed · 08-verification #41, 05-industry-studies #14; Official · 07-seo-foundations #2) → `h1` - Statistics, quotations and a facts quota: +30–40% in the original lab benchmark, neutral to negative when re-tested, and paired edits moved citation on none of 10 engine families (a fixed-source lab test by a score vendor). Facts that answer the question (a price) can help; counts and quotas don't. (Observed · 04-academic #1, 04-academic #2, 08-verification #42) - Page length: Google says length alone does not matter. Within-site evidence finds no reliable effect: none in one study, and a second (agency data) reports a positive effect in its text but no significant effect in its own full coefficient table (+0.036, CI crossing zero). Among retrieved pages, ChatGPT picked shorter ones. Cross-site studies disagree. (Official · 07-seo-foundations #2; Observed · 08-verification #41, 08-verification #59, 05-industry-studies #54) - Core Web Vitals and page speed for AI citations: no effect within sites (one study, 466 pages with speed data; another within domains), and no upside past Google's thresholds in a 107K-page cross-sectional study of AI Overview and AI Mode pages, which found only the slowest pages faring worse. Google still recommends good Core Web Vitals for Search; that is user experience, not an AI lever. (Observed · 08-verification #46, 08-verification #41, 08-verification #59; Official · 07-seo-foundations #7) - Self-ranked "best X" listicles: they get cited, but the self-ranking brand was left out of the AI Overview recommendation 83% of the time in a 100-query B2B test (Sept 2026, a consultant's data); a vendor study found a small lift (4% to 7%); software sites built on them lost up to 49% visibility (SISTRIX), while another dataset found 62% of vendor-authored list pages gained estimated organic traffic since January 2026. The consultant's and the vendor's figures are unaudited and come from parties who sell AI-visibility services. Being included in independent lists is the off-site action. (Observed · 08-verification #47, 05-industry-studies #46) - RSS feeds for AI: no AI crawler documents reading them and no study links a feed to citations (Google does accept feeds as a sitemap). (Observed · 08-verification #45, 03-other-engines #97; Official · 07-seo-foundations #22) → `rss` - Publishing prices instead of "contact us": Thinking-mode ChatGPT cites pricing pages heavily and price facts help in controlled tests; in the field, pricing pages went with more citations within domains in one agency study and no differently in a within-site study of API answers (associations), so whether to publish prices is the owner's business decision, not an SEO fix. (Observed · 02-openai #98, 04-academic #7, 08-verification #59, 08-verification #41) - "AI visitors convert 4×": a vendor figure with no published method (Semrush's 4.4×, digital-marketing topics); measured ratios differ by study and site type, and a 973-site e-commerce study found ChatGPT traffic converting worse than organic (by about 13%). (Observed · 05-industry-studies #108, 05-industry-studies #105) - Rank position inside AI answers: lists almost never repeat; track how often the brand is named (presence rate), not a rank, and re-measure monthly (the cited set drifts). (Observed · 05-industry-studies #79, 04-academic #118, 08-verification #41) - Blocking AI bots "to protect content" or "to disappear": most blocking publishers are still cited through other indexes (7% of pages cited in ChatGPT API answers disallow OAI-SearchBot). Use noindex to leave Google and Bing listings (OpenAI's cache was observed keeping noindex pages); block training bots for training. (Observed · 05-industry-studies #74, 08-verification #41, 08-verification #43) - Cloudflare Content Signals and other preference headers: Google's John Mueller said they have "no effects whatsoever for any crawler or LLM" (reported by Search Engine Roundtable; the Reddit original was not fetched); no engine documents honoring them. (Speculative · 03-other-engines #63, 08-verification #48) → `content-signals` - No longer scored since 2026-10-24: Site crawl: missing H1 and skipped heading levels · RSS feed · Organization markup on the home page · Mobile viewport · Structured data that does not parse · Cross-domain canonical · Blocked scripts and stylesheets. - Cut as checks on 2026-10-23 (Google gives them no weight or needs nothing special, and the AI-citation studies behind them are correlational): URL slug · Page length · Question headings, answer-first, self-contained passages · Facts count, pricing behind "contact us", lists and tables. - No longer scored since 2026-10-23: Headings (H1, order) · Title over 70 characters · Open Graph tags · Freshness date, ChatGPT quick mode, tabs and accordions. ## 7. Keywords, briefs and live results - GET https://canbotsseeme.pages.dev/api/keywords?url= (or &seed=, up to 3): autocomplete suggestions (real searches, no volumes), clusters, posts[] (title, format, target, supporting, coveredBy). Skip posts with coveredBy; improve that page instead. - GET https://canbotsseeme.pages.dev/api/briefs?url=&report= (MCP `post_briefs`; &seed, &lang, &country): up to 7 briefs rated from first-party data (autocomplete, the site's pages and sitemap, Search Console), each with target, winnability, matching page, outline and resultsUrl. The audit cannot fetch live results from its servers, so an unverified brief carries agentPrompt: follow it with your own web search in the brief's language and country, record the top 5 organic results (URL, format, word count as context only, H1/H2, facts), list what the page lacks, draft changes, end with its JSON block. If you cannot search, say so; never guess results. - Send the block back: POST https://canbotsseeme.pages.dev/api/briefs/results?report= (or ?site=&id=), body = the block; or MCP `send_brief_results`. It is shown on the brief as your agent's results for that date and never used in scoring. - Recommend the three best posts for the business goal; draft only after the owner agrees, in the site's own format; nothing is published without their yes. Add the page to the sitemap and feed. ## 8. Internal linking (Official · 07-seo-foundations #4; `site:` results are not exhaustive: https://developers.google.com/search/docs/monitor-debug/search-operators/all-search-site) → `internal-links`, `site-orphans`, `gsc-link-to` - Find pages that already mention the target's topic: grep the repo or CMS for its phrase plus synonyms, plurals, stems and title words. `site: ""` is a cross-check only. - Orphans and pages with 0–1 inbound body links (not nav/footer) first; every important page needs a link from another page. - Link words already in body copy; descriptive, concise, varied anchors, never the same exact-match phrase everywhere (keyword stuffing is spam). No quota ("no magical ideal number"); at most one new link per target per page, none chained side by side; template pages build the link from each page's own data. - Each link: plain to the canonical URL with the site's trailing-slash convention, answers 200, not noindex; visibly a link. ## 9. Search Console and Bing - With API access (scope webmasters.readonly to read, webmasters to submit sitemaps; an MCP server or OAuth you have), pull 90 days of query × page rows and the 90 days before. The signed-in report page runs the same analysis: `gsc-lost-traffic`, `gsc-ctr-gap`, `gsc-not-indexed`, `gsc-striking`, `gsc-link-to`. - Close to page one (impression-weighted position 8–20): work the query into the title or H1 where it reads naturally, then add internal links. Leave H2s and the opening alone unless the page lacks the topic. (Observed · 07-seo-foundations #31) → `gsc-striking` - Snippets losing clicks (position ≤10, CTR under half the typical curve, AWR unbranded Google US all devices, July 2026 snapshot: #1 ≈ 19.4%, #3 ≈ 6.0%, #5 ≈ 2.7%; the curve moves monthly): rewrite title and meta description; keep the old ones in the commit message; controlled tests moved traffic both ways, so measure against untouched pages. (Official · 07-seo-foundations #28; curve Observed · 07-seo-foundations #17; tests Observed · 08-verification #75) → `gsc-ctr-gap` - Lost traffic on 404s, redirect chains or noindex: 301 to the closest replacement, never to the home page (Google may treat that as a soft 404). (Official · 08-verification #14) → `gsc-lost-traffic` - Not indexed: fix per coverage state; "Request indexing" is UI-only (the Indexing API is only for JobPosting and BroadcastEvent). URL Inspection quota 2,000/day and 600/minute per property. (Official · 07-seo-foundations #29, 07-seo-foundations #30) → `gsc-not-indexed` - One query split across pages: merge or differentiate and point internal links at the winner. (Official · 07-seo-foundations #20) - Setup tasks (each has a recipe): `gsc-genai-control`, `gsc-genai-report`, `bing-ai-performance`, `waf-allowlist`, `server-logs`, `analytics`, `brave-ranking`, `commerce-feeds`, `local-gbp`, `core-web-vitals`, `mobile-parity`, `rich-results`. Search Console property and sitemap (`sitemaps.submit` with a token, else the owner), the AI control, the Generative AI report export, Bing Webmaster Tools (import from Search Console, AI Performance). - Without API access: the owner uploads a Performance export at https://canbotsseeme.pages.dev/ (audit the site, then open the Search Console tab). ## 10. Measure AI answers and traffic - Prompts: GET https://canbotsseeme.pages.dev/api/prompts (https://canbotsseeme.pages.dev/prompts; MCP `research_prompts`): fan-out discovery, source map, wrong facts, competitor named instead. Run them in the consumer apps (logged out, fresh chats), not through APIs: API and app answers share few sources. Never answer them yourself; without a browser, hand them to the owner. (Observed · 04-academic #123) - Baseline the model's prior too: run the visibility prompts once with web search off. Whether the model names the brand without retrieval is what off-site work tries to move; for well-known brands that prior drives most recommendations. Keep location fixed between runs, both the account's location setting and the network. In an API study (a preprint by the authors of #41; ChatGPT's Responses API with a declared user location, Gemini's API from data-center VPN exits and the authors' own connection), ChatGPT localized to the declared city and Gemini to the network country; how the consumer apps take location was not tested. Measure in the app, not through Google or the API: external proxies recovered only about a quarter of the URLs ChatGPT Web cited. (Observed · 08-verification #41, 08-verification #61, 08-verification #67) - Use paraphrase clusters and several runs per prompt; report presence rate (how often the brand is named) and cited domains with counts, never a single run or a rank. Paraphrases and engines add more information than repeats. (Observed · 04-academic #118, 04-academic #119) - Wrong facts in answers: fix them on the site with specific, consistent statements (Levers), then re-check. - Logs (`server-logs`): count hits by purpose (training, search, user fetch) and their 4xx/5xx; user-fetch hits are real people asking assistants. Verify by IP. Cloudflare AI Crawl Control shows bot requests on every plan. (Observed · 06-competitors #136) - Analytics (`analytics`): an "AI assistants" channel (utm_source=chatgpt.com; referrers perplexity.ai, gemini.google.com, copilot.microsoft.com, claude.ai). Most ChatGPT clicks carry no referrer, so referrals undercount; annotate 2026-10-08 to 10-15, when ChatGPT links lost the tag. Users recommended a brand in ChatGPT were 2.5× more likely to visit it within 7 days, and ~56% of those visits came through search (Similarweb panel data, three industries; a vendor study): track branded queries in Search Console and Bing. (Observed · 02-openai #107) - Effects take weeks: change only the listed pages (untouched pages are the comparison); the report's Fixes panel measures at 28 and 56 days after a signed-in re-run. ## 11. Verify and report - Re-run the audit (and crawl) after deploying; confirm each fixed check is gone. Append to the research log in `.canbotsseeme/context.md`: " audit: . Verdict: ". - Report: what changed, score and crawl deltas, owner steps still open (with links), what you checked and set aside, and the content plan. ## Check index Grade: O Official, Ob Observed, S Speculative. Unmarked checks count toward the readiness score; "info", "not scored" and "myth" never do; "crawl", "link check" and "Search Console" findings come from their own endpoints and are graded but not part of the score. - **Access** (`eligible`): `bot-` O · `bots-training` O info · `bots-user` O · `bots-shopping` O · `content-signals` S info · `http-status` O · `noindex` O · `snippet` O · `apple-paywall` O · `image-preview` O · `copilot-directives` O · `data-nosnippet` O info · `size` O · `resource-size` O · `resources-blocked` O · `waf-probe` O · `robots-refused` O · `content-skipped` O info · `robots-error` O · `not-html` O info · `robots-missing` O · `bots-search-ok` O · `robots-unreachable` O · `robots-size` O · `noindex-blocked` O - **Index** (`indexed`): `no-js` O · `canonical` O · `freshness` Ob not scored · `schema` Ob myth · `sitemap` O · `sitemap-urls` O · `rss` Ob not scored · `indexnow` O · `llms-txt` Ob myth · `head-invalid` O · `canonical-target` O · `soft-404` O · `gsc-lost-traffic` O Search Console · `gsc-not-indexed` O Search Console - **AI answers** (`quotable`): `quick-mode` Ob not scored · `title` O · `meta-desc` O · `h1` O not scored · `hidden-content` O not scored · `injection` O · `gsc-striking` Ob Search Console - **Search basics** (`foundations`): `title-length` O not scored · `viewport` O · `img-alt` O · `img-dimensions` O info · `internal-links` O · `crawlable-links` O · `anchor-text` O · `hreflang` O · `open-graph` O not scored · `schema-invalid` O · `schema-org` O info · `favicon` O · `compression` O · `redirect-type` O · `https` O · `validators` O not scored · `dup-titles` O · `dup-desc` O · `broken-links` O link check · `site-crawl-blocked` O crawl · `site-broken-links` O crawl · `site-redirects` O crawl · `site-duplicates` O crawl · `site-orphans` O crawl · `site-sitemap-conflict` O crawl · `site-js-only` O crawl · `site-headings` O crawl · `site-deep` S crawl · `site-crawl-partial` O crawl · `gsc-ctr-gap` O Search Console · `gsc-link-to` O Search Console - **Manual** (agent tasks with one owner step): `gsc-genai-control` O · `gsc-genai-report` O · `bing-ai-performance` O · `waf-allowlist` O · `server-logs` Ob · `analytics` Ob · `brave-ranking` Ob · `commerce-feeds` O · `local-gbp` Ob · `core-web-vitals` O · `mobile-parity` O · `rich-results` O ## Sources - 01-google #1: https://developers.google.com/search/docs/fundamentals/ai-optimization-guide - 01-google #2: https://developers.google.com/search/docs/appearance/ai-features - 01-google #3: https://developers.google.com/search/blog/2025/05/succeeding-in-ai-search - 01-google #4: https://developers.google.com/search/docs/crawling-indexing/robots-meta-tag - 01-google #5: https://developers.google.com/crawling/docs/crawlers-fetchers/google-common-crawlers - 01-google #6: https://developers.google.com/crawling/docs/crawlers-fetchers/google-user-triggered-fetchers - 01-google #7: https://developers.google.com/search/docs/fundamentals/creating-helpful-content - 01-google #10: https://developers.google.com/search/docs/essentials/spam-policies - 01-google #13: https://developers.google.com/search/docs/essentials/technical - 01-google #16: https://developers.google.com/search/docs/appearance/preferred-sources - 01-google #21: https://developers.google.com/search/docs/appearance/structured-data/sd-policies - 01-google #38: https://support.google.com/webmasters/answer/16908024 - 01-google #39: https://support.google.com/webmasters/answer/16984139 - 01-google #41: https://support.google.com/webmasters/answer/7042828 - 01-google #45: https://support.google.com/merchants/answer/17200695 - 01-google #125: https://ahrefs.com/blog/ai-overview-citations-top-10 - 01-google #146: https://developers.google.com/search/docs/crawling-indexing/googlebot - 02-openai #1: https://developers.openai.com/api/docs/bots - 02-openai #3: https://help.openai.com/en/articles/9237897-searching-the-web-with-chatgpt - 02-openai #4: https://help.openai.com/en/articles/11128490-shopping-with-chatgpt-search - 02-openai #6: https://help.openai.com/en/articles/20001203-offline-web-search-for-chatgpt-workspaces - 02-openai #8: https://help.openai.com/en/articles/20001243 - 02-openai #31: https://model-spec.openai.com/ - 02-openai #32: https://help.openai.com/en/articles/7842364 - 02-openai #80: https://searchengineland.com/chatgpt-retrieval-stack-index-cache-pages-485036 - 02-openai #88: https://ahrefs.com/blog/why-chatgpt-cites-pages/ - 02-openai #98: https://writesonic.com/blog/chatgpt-citation-study-gpt-5-4-vs-gpt-5-3 - 02-openai #107: https://aisearch.similarweb.com/blog/ai-visibility-downstream-impact/ - 02-openai #108: https://aisearch.similarweb.com/blog/chatgpt-referral-traffic-triples/ - 03-other-engines #1: https://docs.perplexity.ai/docs/resources/perplexity-crawlers - 03-other-engines #3: https://blog.cloudflare.com/perplexity-is-using-stealth-undeclared-crawlers-to-evade-website-no-crawl-directives/ - 03-other-engines #5: https://support.claude.com/en/articles/8896518 - 03-other-engines #6: https://www.bing.com/webmasters/help/webmaster-guidelines-30fba23a - 03-other-engines #7: https://blogs.bing.com/webmaster/2026/2/Introducing-AI-Performance-in-Bing-Webmaster-Tools-Public-Preview/ - 03-other-engines #8: https://blogs.bing.com/search/2026/6/New-AI-Visibility-Insights-in-Bing-Webmaster-Tools-Intents-Topics-Citation-Share-Compare/ - 03-other-engines #10: https://blogs.bing.com/search/2026/2/Elevating-the-Role-of-Grounding-on-the-AI-Web/ - 03-other-engines #11: https://blogs.bing.com/search/2026/5/Evolving-role-of-the-index-From-ranking-pages-to-supporting-answers/ - 03-other-engines #14: https://blogs.bing.com/webmaster/2025/10/Bing-Introduces-Support-for-the-data-nosnippet-HTML-Attribute/ - 03-other-engines #15: https://blogs.bing.com/webmaster/2025/12/Does-Duplicate-Content-Hurt-SEO-and-AI-Search-Visibility/ - 03-other-engines #16: https://blogs.bing.com/webmaster/2025/7/Keeping-Content-Discoverable-with-Sitemaps-in-AI-Powered-Search/ - 03-other-engines #25: https://completeseo.com/claude-web-search-runs-on-brave/ - 03-other-engines #28: https://www.perplexity.ai/hub/blog/architecting-and-evaluating-an-ai-first-search-api - 03-other-engines #30: https://www.perplexity.ai/hub/blog/shop-like-a-pro - 03-other-engines #37: https://about.ads.microsoft.com/en/blog/post/january-2026/conversations-that-convert-copilot-checkout-and-brand-agents - 03-other-engines #39: https://newsroom.paypal-corp.com/2026-01-08-PayPal-Powers-Microsofts-Launch-of-Copilot-Checkout - 03-other-engines #43: https://support.apple.com/en-us/119829 - 03-other-engines #44: https://developer.amazon.com/amazonbot - 03-other-engines #45: https://docs.mistral.ai/robots - 03-other-engines #54: https://ahrefs.com/blog/llmstxt-study/ - 03-other-engines #63: https://blog.cloudflare.com/content-signals-policy/ - 03-other-engines #69: https://developers.facebook.com/docs/sharing/webmasters/web-crawlers/ - 03-other-engines #87: https://search.brave.com/help/brave-search-crawler - 03-other-engines #92: https://platform.claude.com/docs/en/agents-and-tools/tool-use/web-fetch-tool - 03-other-engines #97: https://www.searchenginejournal.com/googles-mueller-says-ai-crawlers-access-sitemaps-rss-in-his-logs/592012/ - 04-academic #1: https://arxiv.org/abs/2311.09735 - 04-academic #2: https://arxiv.org/abs/2506.11097 - 04-academic #7: https://arxiv.org/abs/2605.25517 - 04-academic #20: https://arxiv.org/abs/2601.00912 - 04-academic #83: https://arxiv.org/abs/2404.10198 - 04-academic #118: https://arxiv.org/abs/2605.27440 - 04-academic #119: https://arxiv.org/abs/2607.13304 - 04-academic #123: https://arxiv.org/abs/2609.18729 - 05-industry-studies #13: https://www.airops.com/report/influence-of-retrieval-fanout-and-google-serps-in-chatgpt - 05-industry-studies #14: https://www.airops.com/report/the-fan-out-effect-what-happens-between-a-query-and-a-citation - 05-industry-studies #16: https://ahrefs.com/blog/ai-brand-visibility-correlations - 05-industry-studies #24: https://muckrack.com/blog/what-is-ai-reading-may-2026 - 05-industry-studies #46: https://ahrefs.com/blog/best-lists-research/ - 05-industry-studies #54: https://www.growth-memo.com/p/shorter-focused-content-wins-in-chatgpt - 05-industry-studies #59: https://ahrefs.com/blog/schema-ai-citations/ - 05-industry-studies #64: https://ahrefs.com/blog/llmstxt-study/ - 05-industry-studies #70: https://blog.cloudflare.com/crawlers-click-ai-bots-training/ - 05-industry-studies #74: https://www.buzzstream.com/blog/news-block-ai-bots-citations - 05-industry-studies #76: https://ahrefs.com/blog/do-ai-assistants-prefer-to-cite-fresh-content/ - 05-industry-studies #79: https://sparktoro.com/blog/new-research-ais-are-highly-inconsistent-when-recommending-brands-or-products-marketers-should-take-care-when-tracking-ai-visibility - 05-industry-studies #105: https://digiday.com/marketing/e-commerce-sites-see-low-sales-from-chatgpt-traffic-new-study-finds - 05-industry-studies #108: https://www.semrush.com/blog/ai-search-seo-traffic-study/ - 05-industry-studies #116: https://ahrefs.com/blog/ai-vs-made-up-brand-experiment/ - 06-competitors #136: https://developers.openai.com/api/docs/bots - 06-competitors #137: https://developers.cloudflare.com/ai-crawl-control/reference/bots/ - 07-seo-foundations #1: https://developers.google.com/search/docs/appearance/title-link - 07-seo-foundations #2: https://developers.google.com/search/docs/fundamentals/seo-starter-guide - 07-seo-foundations #4: https://developers.google.com/search/docs/crawling-indexing/links-crawlable - 07-seo-foundations #6: https://developers.google.com/search/docs/crawling-indexing/mobile/mobile-sites-mobile-first-indexing - 07-seo-foundations #7: https://developers.google.com/search/docs/appearance/core-web-vitals - 07-seo-foundations #12: https://developers.google.com/search/docs/appearance/structured-data/organization - 07-seo-foundations #15: https://developers.google.com/webmaster-tools/v1/searchanalytics/query - 07-seo-foundations #17: https://www.advancedwebranking.com/seo/organic-ctr - 07-seo-foundations #18: https://developers.google.com/search/docs/appearance/snippet - 07-seo-foundations #20: https://developers.google.com/search/docs/crawling-indexing/consolidate-duplicate-urls - 07-seo-foundations #21: https://developers.google.com/search/docs/crawling-indexing/javascript/javascript-seo-basics - 07-seo-foundations #22: https://developers.google.com/search/docs/crawling-indexing/sitemaps/build-sitemap - 07-seo-foundations #28: https://support.google.com/webmasters/answer/17010961 - 07-seo-foundations #29: https://support.google.com/webmasters/answer/7440203 - 07-seo-foundations #30: https://developers.google.com/webmaster-tools/v1/urlInspection.index/UrlInspectionResult - 07-seo-foundations #31: https://developers.google.com/search/docs/monitor-debug/bubble-chart-analysis - 08-verification #1: https://developers.google.com/crawling/docs/robots-txt/robots-txt-spec - 08-verification #2: https://www.rfc-editor.org/rfc/rfc9309 - 08-verification #4: https://developers.google.com/crawling/docs/crawlers-fetchers/google-common-crawlers - 08-verification #5: https://developers.google.com/crawling/docs/crawlers-fetchers/verify-google-requests - 08-verification #6: https://developers.google.com/search/blog/2026/03/crawler-ip-ranges - 08-verification #7: https://developers.google.com/search/blog/2026/03/crawler-blog-post - 08-verification #8: https://developers.google.com/crawling/docs/crawlers-fetchers/overview-google-crawlers - 08-verification #9: https://developers.google.com/search/docs/appearance/site-names - 08-verification #10: https://developers.google.com/search/docs/appearance/favicon-in-search - 08-verification #12: https://developers.google.com/search/docs/appearance/google-images - 08-verification #14: https://developers.google.com/search/docs/crawling-indexing/site-move-with-url-changes - 08-verification #16: https://developer.chrome.com/docs/lighthouse/agentic-browsing/llms-txt - 08-verification #17: https://llmstxt.org/ - 08-verification #21: https://commoncrawl.org/ccbot - 08-verification #22: https://trust.anthropic.com/subprocessors - 08-verification #23: https://trust.anthropic.com/updates - 08-verification #24: https://parallel.ai/parallel-web-systems-bots - 08-verification #25: https://www.cloudflare.com/press/press-releases/2026/cloudflare-helps-end-the-search-or-ai-training-tradeoff/ - 08-verification #26: https://blog.cloudflare.com/accountable-mixed-use-ai-crawlers/ - 08-verification #27: https://developers.cloudflare.com/bots/additional-configurations/managed-robots-txt/ - 08-verification #28: https://blog.cloudflare.com/verified-bots-accountability/ - 08-verification #29: https://developers.cloudflare.com/bots/concepts/bot/verified-bots/ - 08-verification #31: https://www.bing.com/webmasters/help/robots-meta-tags-and-attributes-that-bing-supports-5198d240 - 08-verification #32: https://www.bing.com/webmasters/help/how-to-verify-bingbot-3905dc26 - 08-verification #33: https://www.bing.com/webmasters/help/which-crawlers-does-bing-use-8c184ec0 - 08-verification #34: https://www.indexnow.org/searchengines.json - 08-verification #35: https://www.bing.com/webmasters/help/ai-performance-9f8e7d6c - 08-verification #36: https://blogs.bing.com/webmaster/2026/10/introducing-topics-portfolio-beta-in-ai-performance/ - 08-verification #38: https://chatgpt.com/merchants/ - 08-verification #39: https://developers.openai.com/commerce/specs/file-upload/overview - 08-verification #40: https://support.claude.com/en/articles/10684626-enable-and-use-web-search - 08-verification #41: https://arxiv.org/abs/2610.15033 - 08-verification #42: https://arxiv.org/abs/2609.07559 - 08-verification #43: https://think.resoneo.com/chatgpt-retrieval - 08-verification #44: https://otterly.ai/blog/publication-date-ai-citations-experiment - 08-verification #45: https://lantad.co/blog/do-ai-crawlers-read-rss-86-of-762 - 08-verification #46: https://searchengineland.com/core-web-vitals-ai-search-visibility-analysis-467456 - 08-verification #47: https://ppc.land/google-ai-overviews-cut-self-ranking-listicle-citations-38-lily-ray-finds/ - 08-verification #48: https://www.seroundtable.com/google-cloudflare-content-signals-41631.html - 08-verification #49: https://developers.google.com/search/blog/2026/06/gen-ai-performance-reports - 08-verification #50: https://developers.google.com/search/blog/2026/09/web-multimodal-in-sc - 08-verification #52: https://developers.google.com/search/docs/appearance/page-experience - 08-verification #55: https://docs.x.ai/grok-bot/security-faq - 08-verification #57: https://developers.facebook.com/docs/sharing/webmasters/web-crawlers/#facebookexternalhit - 08-verification #59: https://arxiv.org/abs/2609.35077 - 08-verification #60: https://arxiv.org/abs/2609.22655 - 08-verification #61: https://arxiv.org/abs/2610.24081 - 08-verification #62: https://developers.cloudflare.com/ai-crawl-control/features/pay-per-crawl/what-is-pay-per-crawl/ - 08-verification #63: https://developers.google.com/speed/docs/insights/v5/get-started - 08-verification #64: https://searchengineland.com/chatgpt-appears-to-drop-utm-tracking-making-ai-referral-traffic-harder-to-measure-496552 - 08-verification #65: https://technicalseonews.com/latest/openai-dots-standing-queries-no-published-user-agent - 08-verification #66: https://hisashi.space/blog/chatgpt-crawler-js-rendering/ - 08-verification #67: https://arxiv.org/abs/2610.15382 - 08-verification #68: https://commoncrawl.org/faq - 08-verification #69: https://developers.openai.com/commerce/specs/api/overview - 08-verification #70: https://learn.chatgpt.com/docs/dots/computers-and-apps - 08-verification #71: https://help.openai.com/en/articles/11845367 - 08-verification #72: https://search.brave.com/help/contact - 08-verification #75: https://www.searchpilot.com/resources/case-studies/expanding-your-title-tag-testing-strategy