What is a robots.txt file?

A robots.txt file is a text file placed in the root directory of your website that tells search engine crawlers which pages or sections they can or cannot access. It follows the Robots Exclusion Protocol (REP) and is respected by major search engines like Google and Bing.

How do I use the generated robots.txt file?

After generating your robots.txt file, download it and upload it to the root directory of your website (e.g., https://yourdomain.com/robots.txt). Make sure the file is publicly accessible so search engine crawlers can read it.

What does the * (asterisk) user-agent mean?

The asterisk (*) is a wildcard that applies rules to all web crawlers and bots that haven't been given specific instructions. It's commonly used as a catch-all rule when you want the same behavior for all crawlers.

How do I block AI bots from crawling my site?

Use the 'Block AI Bots' preset which adds Disallow: / rules for GPTBot, ChatGPT-User, CCBot, anthropic-ai, Claude-Web, and Google-Extended. These are the main AI training crawlers used by OpenAI, Anthropic, and Google.

Does robots.txt prevent content from being indexed?

No — robots.txt prevents crawlers from accessing pages, but it does not guarantee content won't be indexed. Search engines may still index a page if other pages link to it, even if crawling is disallowed. Use the noindex meta tag or X-Robots-Tag header to prevent indexing.

What is a Sitemap directive in robots.txt?

The Sitemap directive in robots.txt tells crawlers where to find your XML sitemap. For example: Sitemap: https://example.com/sitemap.xml. You can include multiple Sitemap lines if you have more than one sitemap.

What is crawl-delay and should I use it?

The Crawl-delay directive tells crawlers how many seconds to wait between requests. This can help reduce server load. However, Google ignores this directive — use Google Search Console to set Googlebot's crawl rate instead.

Free SEO Tool

Generate Perfect
robots.txt Files

Create, validate, and export robots.txt files for your website in seconds. Add user-agents, crawl rules, sitemap URLs, and more — all in your browser.

🤖

Multi User-Agent Support

Add rules for Googlebot, Bingbot, AI crawlers, or any custom user-agent with ease.

⚡

Instant Preview

See your robots.txt file update live as you build it. Copy or download in one click.

🛡️

Smart Presets

Start fast with presets: Allow All, Block All, Block AI Bots, or Standard SEO configuration.

robots.txt Generator

Quick Presets

User-Agent Rules

Allow

Disallow

None

Crawl-delay (seconds)

Sitemap URLs

No sitemaps added

Host Directive

Live Previewrobots.txt

User-agent: *
Allow: /

Upload robots.txt to the root of your website (e.g. https://example.com/robots.txt)

Frequently Asked Questions

Everything you need to know about Robots.txt Generator.

Generate Perfectrobots.txt Files

Multi User-Agent Support

Instant Preview

Smart Presets

Frequently Asked Questions

Generate Perfect
robots.txt Files