Sitemap: https://rate.house/sitemap.xml User-agent: SemrushBot Disallow: / User-agent: AhrefsBot Disallow: / # ====================== # AI TRAINING BOTS # (Blocked — no SEO impact) # ====================== # OpenAI (training only) User-agent: GPTBot Disallow: / # Anthropic User-agent: anthropic-ai Disallow: / User-agent: ClaudeBot Disallow: / User-agent: Claude-Web Disallow: / # Google AI training (does NOT affect Google Search) User-agent: Google-Extended Disallow: / # Apple AI training (does NOT affect Siri/search) User-agent: Applebot-Extended Disallow: / # Meta AI training (does NOT affect Facebook/Instagram link previews) User-agent: Meta-ExternalAgent Disallow: / User-agent: Meta-ExternalFetcher Disallow: / # Common Crawl (major AI training data source) User-agent: CCBot Disallow: / # Cohere User-agent: cohere-ai Disallow: / # ByteDance / TikTok User-agent: Bytespider Disallow: / # Diffbot User-agent: Diffbot Disallow: / # AI2 (Allen Institute for AI) User-agent: AI2Bot Disallow: / User-agent: Ai2Bot-Dolma Disallow: / # Webz.io User-agent: Webzio-Extended Disallow: / # Omgili User-agent: Omgilibot Disallow: / # Timpi User-agent: Timpibot Disallow: / # ImagesiftBot User-agent: ImagesiftBot Disallow: / # iask User-agent: iaskspider Disallow: / # Scrapy (generic scraping framework) User-agent: Scrapy Disallow: / # ====================== # AI SEARCH & ASSISTANT BOTS # (Explicitly allowed — these drive # citations/referrals in AI answers, # not model training) # ====================== # OpenAI search index & user-initiated fetches User-agent: OAI-SearchBot Disallow: /toggle-theme Allow: / User-agent: ChatGPT-User Disallow: /toggle-theme Allow: / # Anthropic user-initiated fetches & search index User-agent: Claude-User Disallow: /toggle-theme Allow: / User-agent: Claude-SearchBot Disallow: /toggle-theme Allow: / # Perplexity search index & user-initiated fetches User-agent: PerplexityBot Disallow: /toggle-theme Allow: / User-agent: Perplexity-User Disallow: /toggle-theme Allow: / # Google Gemini/AI features grounded in Google Search # (Googlebot itself is allowed by default below) # DuckDuckGo AI assist User-agent: DuckAssistBot Disallow: /toggle-theme Allow: / # Mistral user-initiated fetches User-agent: MistralAI-User Disallow: /toggle-theme Allow: / # ====================== # EVERYTHING ELSE ALLOWED # (Search engines, social previews, # AI search, discovery bots) # ====================== User-agent: * Disallow: /toggle-theme Allow: /