robots.txt is a text file in a website's root directory that tells search engine crawlers which pages they can or cannot access. It follows the Robots Exclusion Protocol standard.

How do I test if a URL is blocked by robots.txt?

Enter the website base URL (to fetch its robots.txt), the URL path you want to test, and the user-agent (e.g. Googlebot). The tester parses the robots.txt directives and tells you if the path is allowed or disallowed.

Can I test with custom user-agents?

Yes. You can select Googlebot, Bingbot, or type any custom user-agent name to test how that crawler sees the robots.txt rules.

🤖 Robots.txt Tester

Name: Robots.txt Tester
Author: ToolStand

Test any URL against robots.txt rules — see what bots can and cannot crawl

📌 Embed This Tool

Add the Robots.txt Tester to your website for free.

<!-- Robots.txt Tester — Embed by ToolStand --><div style="position:relative;width:100%;max-width:100%;overflow:hidden;"><iframe src="https://toolstand.io/tools/robots-tester/embed.html" width="100%" height="600" frameborder="0" style="border:1px solid #e2e8f0;border-radius:8px;background:#fff;" loading="lazy" title="Robots.txt Tester | ToolStand" allow="clipboard-write" sandbox="allow-scripts allow-same-origin allow-popups allow-forms"></iframe></div>

What Is the Robots.txt Tester?

Robots.txt Tester fetches a website's robots.txt file and tests whether a specific URL path is allowed or blocked for a given user-agent (Googlebot, Bingbot, etc.). Essential for SEO auditing and crawl budget optimization.

Key Features

Fetch robots.txt: Retrieves the live robots.txt from any website.
Rule Parsing: Parses Allow/Disallow directives for each user-agent.
Path Testing: Test any URL path against the parsed rules.
Custom User-Agents: Choose Googlebot, Bingbot, wildcard, or a custom bot.

📋 When to Use

Use to verify that important pages are crawlable, check that staging sites are properly blocked, debug unexpected deindexing, and audit robots.txt before deployments.

⚙️ How It Works

The tool fetches /robots.txt via the ToolStand proxy, parses the directives line-by-line, groups rules by user-agent, then matches the test path against the most specific rule for the selected user-agent.

How to Use the Robots.txt Tester

Enter a base URL to fetch its robots.txt.
Enter a path you want to test (e.g. /admin/).
Select a user-agent or type a custom one.
Click Test to see the result and raw robots.txt.

Frequently Asked Questions

What is robots.txt?

A text file at the root of a website that instructs crawlers which paths they may or may not access.

How do I test if a URL is blocked?

Enter the site's base URL, the path, and a user-agent. The tester fetches robots.txt and applies the rules.

What does Disallow: / mean?

It blocks the user-agent from crawling the entire site.

Can I use custom user-agents?

Yes, select "Custom…" and type any bot name.

Is it free?

Yes, completely free — no sign-up and no API keys. A fair-use rate limit of 30 checks per minute per IP applies to protect the service.