User-agent: AwarioBot Disallow: / User-agent: GPTBot Disallow: / Allow: /crawl User-agent: ClaudeBot Disallow: / Allow: /crawl User-agent: anthropic-ai Disallow: / Allow: /crawl User-agent: CCBot Disallow: / Allow: /crawl User-agent: meta-externalagent Disallow: / Allow: /crawl User-agent: FacebookBot Disallow: / Allow: /crawl User-agent: Bytespider Disallow: / Allow: /crawl User-agent: Applebot-Extended Disallow: / Allow: /crawl User-agent: Amazonbot Disallow: / Allow: /crawl User-agent: AhrefsBot Disallow: / Allow: /crawl User-agent: SemrushBot Disallow: / Allow: /crawl User-agent: MJ12bot Disallow: / Allow: /crawl User-agent: DataForSeoBot Disallow: / Allow: /crawl User-agent: DotBot Disallow: / Allow: /crawl User-agent: BLEXBot Disallow: / Allow: /crawl User-agent: serpstatbot Disallow: / Allow: /crawl User-agent: Barkrowler Disallow: / Allow: /crawl User-agent: OAI-SearchBot Allow: / Disallow: /login Disallow: /signup Disallow: /auth/ Disallow: /api/ Disallow: /settings Disallow: /sources/new Disallow: /feeds/new User-agent: ChatGPT-User Allow: / Disallow: /login Disallow: /signup Disallow: /auth/ Disallow: /api/ Disallow: /settings Disallow: /sources/new Disallow: /feeds/new User-agent: Claude-SearchBot Allow: / Disallow: /login Disallow: /signup Disallow: /auth/ Disallow: /api/ Disallow: /settings Disallow: /sources/new Disallow: /feeds/new User-agent: Claude-User Allow: / Disallow: /login Disallow: /signup Disallow: /auth/ Disallow: /api/ Disallow: /settings Disallow: /sources/new Disallow: /feeds/new User-agent: PerplexityBot Allow: / Disallow: /login Disallow: /signup Disallow: /auth/ Disallow: /api/ Disallow: /settings Disallow: /sources/new Disallow: /feeds/new User-agent: Perplexity-User Allow: / Disallow: /login Disallow: /signup Disallow: /auth/ Disallow: /api/ Disallow: /settings Disallow: /sources/new Disallow: /feeds/new User-agent: Google-Extended Allow: / Disallow: /login Disallow: /signup Disallow: /auth/ Disallow: /api/ Disallow: /settings Disallow: /sources/new Disallow: /feeds/new User-agent: Bingbot Allow: / Disallow: /login Disallow: /signup Disallow: /auth/ Disallow: /api/ Disallow: /settings Disallow: /sources/new Disallow: /feeds/new User-agent: Meta-ExternalFetcher Allow: / Disallow: /login Disallow: /signup Disallow: /auth/ Disallow: /api/ Disallow: /settings Disallow: /sources/new Disallow: /feeds/new User-agent: DuckAssistBot Allow: / Disallow: /login Disallow: /signup Disallow: /auth/ Disallow: /api/ Disallow: /settings Disallow: /sources/new Disallow: /feeds/new User-agent: MistralAI-User Allow: / Disallow: /login Disallow: /signup Disallow: /auth/ Disallow: /api/ Disallow: /settings Disallow: /sources/new Disallow: /feeds/new User-agent: meta-webindexer Allow: / Disallow: /login Disallow: /signup Disallow: /auth/ Disallow: /api/ Disallow: /settings Disallow: /sources/new Disallow: /feeds/new User-agent: YouBot Allow: / Disallow: /login Disallow: /signup Disallow: /auth/ Disallow: /api/ Disallow: /settings Disallow: /sources/new Disallow: /feeds/new User-agent: ExaSearchBot Allow: / Disallow: /login Disallow: /signup Disallow: /auth/ Disallow: /api/ Disallow: /settings Disallow: /sources/new Disallow: /feeds/new User-agent: AionBot Allow: / Disallow: /login Disallow: /signup Disallow: /auth/ Disallow: /api/ Disallow: /settings Disallow: /sources/new Disallow: /feeds/new User-agent: * Allow: / Disallow: /login Disallow: /signup Disallow: /auth/ Disallow: /api/ Disallow: /settings Disallow: /sources/new Disallow: /feeds/new Sitemap: https://nichedb.dev/sitemap.xml Sitemap: https://nichedb.dev/sitemap.xml