Robots.txt Generator
SEO toolNewGenerate a robots.txt file for your website.
This tool runs entirely in your browser. Your files never leave your device — nothing is uploaded.
Crawler rules
Use * to apply the rules to every crawler.
One path per line. Leave blank if you have nothing to allow.
One path per line. These paths are blocked from crawling.
User-agent: * Disallow: /admin
Frequently asked questions
What does a robots.txt file do?
A robots.txt file tells search engine crawlers which parts of your site they may or may not request. You set a User-agent and then Allow or Disallow specific paths, and can point crawlers to your sitemap. It guides well-behaved bots but is not a security control — anything you truly need hidden should be protected another way.
Is my data uploaded anywhere when I use this generator?
No. The entire robots.txt file is built right in your browser with client-side JavaScript. Your paths, sitemap URL, and settings never leave your device and nothing is sent to a server, so it stays completely private.
What is Crawl-delay and should I set it?
Crawl-delay asks crawlers to wait the given number of seconds between requests, which can ease load on smaller servers. It is optional and not honored by every crawler (Google ignores it, for example). Leave it blank unless you have a specific reason to throttle bots.