🤖 Robots.txt Generator
🔍 Robots.txt Generator for SEO
A robots.txt file tells compliant search engine crawlers which URLs or website paths they may crawl. This free Robots.txt Generator lets you create a structured file using User-agent, Allow, Disallow, Sitemap and optional crawler directives.
Why use robots.txt?
Robots.txt can help manage crawler access to areas that do not need to be crawled, such as administrative directories, internal search pages, shopping cart URLs and other low-value crawl paths. It should be used as a crawl-management mechanism, not as a security system.
Important robots.txt directives
- User-agent: Identifies which crawler a rule group applies to.
- Disallow: Requests that a crawler does not crawl a specified path.
- Allow: Can permit crawling of a path that would otherwise be covered by a Disallow rule, where supported.
- Sitemap: Provides the absolute URL of an XML sitemap to help crawlers discover URLs.
- Crawl-delay: A non-standard directive whose support differs between crawlers.
- Host: A non-standard directive historically associated with Yandex and not required for most websites.
Robots.txt and SEO
Blocking a URL in robots.txt does not guarantee that the URL will be removed from Google's index. If a page must not be indexed, use an appropriate indexing control such as a noindex directive where possible, while ensuring that the crawler can access the page when required.
Common mistakes
- Do not use robots.txt as a replacement for authentication.
- Do not put passwords, secrets or confidential information in it.
- Be careful when blocking CSS, JavaScript or image resources.
- Do not accidentally use Disallow: / on a live site.
- Always check the generated file before publishing it.
How to use this Robots.txt Generator
Select a User-agent, add the paths you want to allow or disallow, enter your sitemap URL if available, and optionally enable the common rules. The robots.txt code updates automatically. Use the Copy button to copy the generated file or Download to save it as robots.txt.
Where should robots.txt be placed?
The file normally belongs at the root of the host, for example: https://example.com/robots.txt. Make sure the file is publicly accessible and returns the expected text content.
