# Northeast Digestive — robots.txt
# https://northeastdigestive.com/
# Last updated: 2026-10-28 (Meta per-crawler rules ahead of Meta's revised
# Commercial Terms, effective 2026-10-30)
#
# Goal: maximum visibility across traditional search engines (Google, Bing),
# AI answer/search engines (ChatGPT, Claude, Perplexity, Gemini, Copilot,
# Meta AI), and AI training crawlers — while keeping non-content utility
# pages out of search and AI results.
#
# This site welcomes all reputable crawlers, including:
#   Googlebot, Bingbot, DuckDuckBot                   (traditional search)
#   OAI-SearchBot, ChatGPT-User, GPTBot               (OpenAI / ChatGPT)
#   Claude-SearchBot, Claude-User, ClaudeBot          (Anthropic / Claude)
#   PerplexityBot, Perplexity-User                    (Perplexity)
#   Google-Extended, Applebot, Applebot-Extended      (Gemini / Apple Intelligence)
#   Amazonbot, Bytespider, CCBot                      (Amazon / ByteDance / Common Crawl)
#   Meta-WebIndexer, Meta-ExternalAds                 (Meta AI search / Meta ads)
#
# One exception: Meta-ExternalAgent (Meta model training) is opted out in
# its own group below. Every other crawler is covered by the wildcard rule.
# Rules are intentionally listed most-specific-first so that even simple,
# order-sensitive parsers apply them correctly.
#
# Named groups: a crawler follows ONLY the most specific group that names
# it and ignores "User-agent: *". The Meta groups therefore repeat the
# wildcard rules. Keep them in sync.
#
# NOTE: Content-Signal is a proposed, not-yet-finalized extension to the
# robots.txt standard (see contentsignals.org). Compliant crawlers that don't
# recognize it will simply ignore the line.
User-agent: *
Content-Signal: search=yes, ai-input=yes, ai-train=yes
# Internal Webflow design/style guide -- not real content, keep out of results
Disallow: /style-page
# Form-confirmation page -- thin/duplicate content, not a useful landing page
Disallow: /thank-you
# Hidden infinite-scroll pagination parameters on the blog feed -- the
# canonical /blog and each individual /blog/[slug] post remain fully crawlable
Disallow: /blog?*
# Everything else is open to all crawlers
Allow: /

# Meta AI search + Meta ads crawler (ads: client decision, allowed by
# default; change that line to its own "Disallow: /" group to opt out)
User-agent: Meta-WebIndexer
User-agent: Meta-ExternalAds
Content-Signal: search=yes, ai-input=yes, ai-train=no
Disallow: /style-page
Disallow: /thank-you
Disallow: /blog?*
Allow: /

# Meta model training: opted out (2026-10-28)
User-agent: Meta-ExternalAgent
Disallow: /

Sitemap: https://northeastdigestive.com/sitemap.xml

Sitemap: https://northeastdigestive.com/sitemap.xml