# Applied Clinical Judgement — robots.txt
# Owned at the origin (Rank Math → General Settings → Edit robots.txt).
# Cloudflare's managed robots.txt must be OFF, or its block is injected above
# this one and you get two conflicting "User-agent: *" groups.
#
# Policy: search and real-time AI retrieval are welcome; AI training is not.
# Training crawlers are listed individually below. Search/retrieval agents
# (Claude-SearchBot, OAI-SearchBot, PerplexityBot) are deliberately NOT listed.
#
# Maintenance: this list no longer auto-updates. Review when new AI crawlers
# appear. Last reviewed 2026-09-07.

User-agent: *
Content-Signal: search=yes, ai-input=yes, ai-train=no, use=reference
Allow: /
Disallow: /wp-admin/
Disallow: /go
Allow: /wp-admin/admin-ajax.php

# Never disallow the signpost surfaces: /llms.txt, /official-apply-links/,
# /apply/*, /r/* — they must stay fetchable by search and retrieval agents.

# --- AI training crawlers: not permitted ----------------------------------

User-agent: Amazonbot
Disallow: /

User-agent: Applebot-Extended
Disallow: /

User-agent: Bytespider
Disallow: /

User-agent: CCBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: GPTBot
Disallow: /

User-agent: meta-externalagent
Disallow: /

User-agent: CloudflareBrowserRenderingCrawler
Disallow: /

Sitemap: https://applied-clinical-judgement.co.uk/sitemap_index.xml