Robots.txt Generator
Generate a correctly formatted robots.txt file, specifying which paths search engines should avoid crawling and where your sitemap lives.
About this robots.txt generator
robots.txt tells search engine crawlers which parts of your site to skip, getting the syntax wrong can accidentally block, or fail to block, the wrong pages. This tool generates clean, correct syntax from a simple list.
How robots.txt generator works
Your list of paths to disallow is formatted into standard robots.txt syntax, along with a sitemap directive pointing search engines to your sitemap file.
Common uses
- Creating a robots.txt file for a new website
- Blocking search engines from crawling admin or private paths
- Pointing search engines to your sitemap correctly
How to use this tool
- List the paths you want to disallow, one per line.
- Enter your sitemap URL.
- Click Generate and copy the result into a robots.txt file at your site's root.
What to know before using it
Generates a straightforward, universal robots.txt. For advanced per-crawler rules (different directives for Googlebot versus other bots), additional manual editing would be needed.
Privacy and browser-first processing
Generated entirely in your browser, nothing is logged.
Frequently asked questions
Where does robots.txt need to be placed?
At the root of your domain, for example yoursite.com/robots.txt, it will not work if placed in a subfolder.
Does robots.txt stop a page from being indexed?
Not reliably on its own, it discourages crawling, but a page can still get indexed if linked from elsewhere, use a noindex meta tag for a guaranteed exclusion from search results.