CyberCodeRobots.txt Generator
Free robots.txt generator. Build a correct robots.txt with allow/disallow rules and a sitemap reference — choose a preset or write custom rules, then download the file.
What does robots.txt do?
robots.txt is a plain text file at the root of your site (example.com/robots.txt) that tells search engine crawlers which paths they may or may not crawl. It is the first file Googlebot requests when visiting a site.
Two things it does not do: it does not remove pages from Google (use a noindex meta tag for that), and it does not provide security — the file is public, so never list secret paths in it. Always include your sitemap URL so crawlers discover your pages faster.
How to use
- 01Pick a preset: allow everything, block everything, or custom rules.
- 02For custom rules, list Disallow and Allow paths one per line.
- 03Add your sitemap URL — strongly recommended for SEO.
- 04Generate and upload the file to your site root as robots.txt.
Frequently asked questions
- Where do I put the robots.txt file?
- At the root of your domain — it must be reachable at https://yourdomain.com/robots.txt. Crawlers do not look for it anywhere else.
- Does Disallow remove a page from Google?
- No — this is the most common robots.txt misconception. Disallow only stops crawling; an already-indexed page can stay in results. To remove a page, allow crawling and add a noindex meta tag instead.
- Can robots.txt protect private content?
- No. The file is public and merely a polite request that good bots follow. Use authentication for anything actually private — and never list secret URLs in robots.txt, as that advertises them.