Generator

Robots.txt Generator

Build a robots.txt file with allow/disallow rules, sitemap, and one-click AI-bot blocking (GPTBot, ClaudeBot, Google-Extended, and more).

Global crawl rules

Allow all crawlers by default

Off = block everything except listed allows

Disallowed paths (one per line)

Explicitly allowed paths (one per line)

Crawl-delay (seconds)

Sitemap URL

Block AI crawlers

Stop these bots from scraping your content for LLM training.

GPTBot (OpenAI)

Crawls for ChatGPT training

ClaudeBot (Anthropic)

Crawls for Claude training

Google-Extended

Crawls for Gemini training (separate from Googlebot)

CCBot (Common Crawl)

Used by many AI training sets

Bytespider (ByteDance)

Crawls for ByteDance/Doubao training

PerplexityBot

Perplexity AI search

anthropic-ai

Anthropic legacy crawler

Applebot-Extended

Apple Intelligence training

robots.txt

# Generated by onlinecorners.com

User-agent: *
Allow: /
Disallow: /admin/
Disallow: /api/
Disallow: /private/

User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: CCBot
Disallow: /

User-agent: Bytespider
Disallow: /

User-agent: PerplexityBot
Disallow: /

User-agent: anthropic-ai
Disallow: /

User-agent: Applebot-Extended
Disallow: /

Important

Robots.txt is a request, not enforcement. Reputable bots respect it; malicious scrapers will ignore it. Pair with server-level blocking (Cloudflare WAF, rate limits) for real protection.