Robots.txt Generator
Generate a robots.txt file with multiple user-agent rules, allow/disallow paths, and a sitemap reference.
User-agent: * Disallow: Sitemap: https://example.com/sitemap.xml
Save the output as a file named robots.txt in the root of your website so it is reachable at https://yoursite.com/robots.txt.
Frequently Asked Questions
What is robots.txt?
robots.txt is a text file placed at the root of your site that tells search engine crawlers which pages they may or may not visit. It is a suggestion, not an access control.
What is a User-agent?
It identifies which crawler the rule applies to. An asterisk (*) applies to all crawlers. You can target specific bots like Googlebot or Bingbot with separate rules.
Does robots.txt block access?
No. It only tells well-behaved crawlers what to avoid. Anyone can still visit a disallowed URL directly. To truly block access, use authentication or noindex directives.
Is my data sent anywhere?
No. The file is generated in your browser. Nothing leaves your device.
Related Tools
Meta Tag Generator
Generate complete HTML meta tags for better SEO and social sharing.
SERP Snippet Preview
Preview how your page title and description will appear in Google results.
Schema Markup Generator
Generate JSON-LD schema markup for articles, FAQs, products, and more.
Keyword Density Checker
Analyze keyword frequency and density in any piece of text or webpage content.