How to Generate a Robots.txt File for Your Website
A robots.txt file sits at the root of your website and tells search engine crawlers which pages they can and cannot access. A misconfigured robots.txt can accidentally hide your entire site from Google or expose admin pages you meant to keep private. A generator helps you get it right the first time.
Robots.txt syntax
The file uses a simple directive format: User-agent specifies which crawler the rules apply to, Disallow blocks specific paths, Allow overrides a broader Disallow for specific sub-paths, and Sitemap points to your XML sitemap URL. The ToolStand Robots.txt Generator provides a visual interface — check boxes for which crawlers to target and which directories to block, and the generator produces the correct syntax.
Common configurations
Allow all crawlers everywhere: User-agent: * and Disallow: (empty) — the most permissive setup. Block all crawlers: User-agent: * and Disallow: / — for development sites. Block specific directories: Disallow: /admin/, Disallow: /api/, etc. Block specific crawlers: Target GPTBot, CCBot, or other AI training crawlers while allowing search engines. Add sitemap: Sitemap: https://yoursite.com/sitemap.xml — helps crawlers discover your content faster.
Pairing with SEO tools
The robots.txt generator works alongside other ToolStand SEO tools. Use the Meta Tag Generator to create page-level meta tags. Use the SERP Preview Tool to see how your pages look in Google results. Together, these tools cover the technical SEO basics without requiring expensive SEO software.
Blocking AI crawlers — GPTBot, CCBot, and the new bot landscape
As of 2026, dozens of AI training crawlers actively scrape the web, and many site owners want to block them without blocking search engines. Major AI bots and their user-agents: GPTBot (OpenAI), CCBot (Common Crawl, used by many LLM training datasets), Claude-Web/anthropic-ai (Anthropic), Google-Extended (Google's AI training, separate from Googlebot), Bytespider (TikTok/ByteDance), and Omgilibot/FacebookBot (Meta). You can block all AI crawlers while allowing search engines with separate User-agent blocks: User-agent: GPTBot / Disallow: /. The Robots.txt Generator includes a checkbox for AI crawlers — one click adds blocks for all major AI bots. Important: robots.txt is a request, not an enforcement mechanism. Malicious crawlers ignore it. For true access control, use IP blocking or authentication.
Wildcards, pattern matching, and the $ end-of-line anchor
Robots.txt supports limited pattern matching beyond simple path blocking. Asterisk (*) matches any sequence of characters: Disallow: /*.pdf$ blocks all PDF files. Dollar sign ($) marks end-of-URL: Disallow: /temp$ blocks /temp but not /templates. Without $, Disallow: /temp is a prefix match and blocks /temp, /templates, /temperature, and anything else starting with /temp. This is the most common robots.txt mistake: unintended broad blocking from missing $. The generator adds $ to directory-level rules that are meant to be exact, preventing accidental over-blocking. Always test your robots.txt with the Robots.txt Tester before deploying.
The robots.txt tag equivalent — on-page control
Robots.txt controls crawling at the site level; the robots meta tag controls indexing at the page level. <meta name="robots" content="noindex, nofollow"> in a page's <head> tells search engines not to index the page and not to follow its links — even if robots.txt allows crawling. The difference matters: robots.txt blocks crawling (the page may still appear in search results as a URL-only listing), while the meta tag blocks indexing (the page will not appear in results at all). Use both: robots.txt to manage crawl budget on large sites and meta tags to control indexing of individual pages. The Meta Tag Generator creates these tags with the correct syntax.
Crawl budget — why a good robots.txt saves money on large sites
Google allocates a "crawl budget" — roughly the number of pages it will crawl on your site per day. For sites with millions of pages (e-commerce, news, forums), blocking low-value URLs in robots.txt preserves crawl budget for important pages. Block: faceted navigation URLs (/search?sort=price&page=5), filtered views, printer-friendly versions, internal search results, and staging/development subdomains. The savings compound: blocking 100,000 low-value URLs means 100,000 more crawls for your product pages, blog posts, and category pages. The Robots.txt Generator helps you systematically exclude these patterns.
Robots.txt validation — common errors that deindex your site
The most catastrophic robots.txt error is User-agent: * followed by Disallow: / — this blocks every crawler from every page. This often happens accidentally when developers copy a staging-site configuration to production. Other common errors: (1) trailing slash mismatch: Disallow: /admin blocks /admin but not /admin/ — write Disallow: /admin/ if your admin panel has a trailing slash; (2) case sensitivity: Disallow: /PDF does not block /pdf on case-sensitive servers; (3) missing sitemap directive: without Sitemap: https://example.com/sitemap.xml, crawlers discover pages more slowly. After generating your robots.txt, validate it with Google Search Console's robots.txt Tester, or use the Robots.txt Tester to check specific URLs against your rules.
Frequently Asked Questions
Can I block a specific URL pattern that contains a query parameter?
Yes, but robots.txt pattern matching is prefix-based — it does not support regex. Disallow: /*?sort= blocks any URL containing ?sort= because * matches any characters before the query string. For more complex filtering, use noindex meta tags or X-Robots-Tag HTTP headers, which support finer control.
Do all search engines respect robots.txt?
Major search engines (Google, Bing, Yandex, Baidu, DuckDuckGo) respect robots.txt. Malicious bots, scrapers, and some AI training crawlers ignore it. Robots.txt is a convention, not a security mechanism. For sensitive content, use authentication, IP restrictions, or serve the content behind a login.
Where should the robots.txt file be located?
At the root of your domain: https://example.com/robots.txt. It must be accessible at exactly this path — crawlers will not look for it in subdirectories. The file is publicly visible; anyone can read your robots.txt to see which paths you are blocking.
How do I allow Googlebot but block all other crawlers?
Use two separate User-agent blocks: first User-agent: Googlebot / Disallow: (empty Disallow means allow all), then User-agent: * / Disallow: / (block everything else). Crawlers match the most specific rule, so Googlebot uses the first block and all others use the second. The generator creates this pattern automatically when you select per-bot settings.
What happens if I do not have a robots.txt file at all?
Crawlers assume everything is allowed. Your site will be indexed fully, including pages you might prefer to keep private (admin panels, staging sites, API endpoints). Having no robots.txt is better than having a broken one that accidentally blocks everything — but having a properly configured one is best.
How often should I update my robots.txt?
Review it whenever you add new sections to your site, change your URL structure, or launch a new subdomain. For rapidly changing sites (e-commerce with rotating inventory, news sites), review monthly. For stable sites, a quarterly review with your sitemap update is sufficient.
Explore all 109 free tools at toolstand.io. Free, forever. No sign-up. No download. Just tools that work.