Does robots.txt keep a page out of Google?

No. It prevents crawling, not indexing. A blocked URL that is linked from elsewhere can still be indexed without a snippet. Use a noindex meta tag or header on a crawlable page instead.

Is robots.txt a security measure?

The opposite. The file is public and effectively lists paths you consider sensitive. Protect private content with authentication, never with a Disallow rule.

Which rule wins when Allow and Disallow both match?

The most specific rule, measured by matched path length. If they are equally specific, Allow wins — per RFC 9309 and Google's implementation.

Do I need a separate file for each subdomain?

Yes. robots.txt applies to one origin, so blog.example.com and example.com each need their own file at their own root.

Network & DNS🌐 YAS server-sideAPI availableintermediate

robots.txt Viewer

Fetches and parses a site's robots.txt to show allowed/disallowed paths.

Processed by YAS · not stored

Ready to runInstant execution

All tools →

Site URL

Loading tool…

What does this tool do?

Fetches and parses a site's robots.txt to show allowed/disallowed paths.

Why would I use it?

You want to know why a page is not being indexed.
You are auditing crawl rules before an SEO push.
You want to check if a path is blocked for crawlers.

Real-life example

Input

https://example.com

Output

User-agent: *
Disallow: /private/

Rules are shown per user-agent.

Input → Process → Output → Next

Input: Enter a site URL.
Process: YAS fetches /robots.txt and parses the rules.
Output: Parsed rules per user-agent.
Next action: Adjust rules if legitimate pages are blocked.

Common mistakes

Assuming robots.txt enforces security (it is advisory).
Checking a path that crawlers would still reach via sitemaps.
Forgetting that directives are case-sensitive.

What the result means

The rules tell cooperative crawlers what to avoid.

Privacy & security

Your input is sent to YAS infrastructure because the tool requires server-side processing or public network queries. Input is not stored.

API

Endpoint

POST https://yas.sh/api/v1/tools/robots-txt-viewer

Request Header

Content-Type: application/json

cURL

curl -X POST "https://yas.sh/api/v1/tools/robots-txt-viewer" \
  -H "Content-Type: application/json" \
  -d '{"url":"https://example.com"}'

JavaScript

const res = await fetch("https://yas.sh/api/v1/tools/robots-txt-viewer", {
  method: "POST",
  headers: { "Content-Type": "application/json" },
  body: JSON.stringify({
  "url": "https://example.com"
}),
});
const data = await res.json();

Python

import requests

r = requests.post("https://yas.sh/api/v1/tools/robots-txt-viewer", json={"url":"https://example.com"})
data = r.json()

Field	Type	Required	Description
url	string	Yes	Site URL (origin)

Success response

{ "origin": "https://example.com", "status": 200, "rules": [{ "userAgent": "*", "allow":[], "disallow":["/admin"] }], "sitemaps": [...] }

Fetch and parse a site's robots.txt rules.

Error responses

400 VALIDATION_ERROR — invalid input or unsupported option.
413 PAYLOAD_TOO_LARGE — input exceeds the 64 KB limit.
429 RATE_LIMIT_EXCEEDED — rate limit exceeded (60 req/min).

Limits

Maximum input: 64 KB per request.
Rate limit: 60 requests/min per IP address.
Authenticated accounts benefit from higher tier quotas.

How robots.txt Viewer works

robots.txt is a plain-text file at the origin root that tells crawlers which paths they may request. It is grouped by User-agent, with Disallow and Allow rules whose matching follows the Robots Exclusion Protocol standardised as RFC 9309: the most specific matching rule wins, measured by path length, and Allow beats Disallow at equal specificity. Google supports wildcards (*) and end-of-URL anchors ($), which are extensions rather than part of the original convention.

The critical misunderstanding is what a Disallow actually does. It prevents crawling, not indexing. A disallowed URL that is linked from elsewhere can still appear in search results — without a snippet, because the crawler was never allowed to fetch the content. Removing a page from the index requires a noindex directive on a page the crawler is allowed to fetch, which means it must not be disallowed in robots.txt.

When to use it: real-world scenarios

Diagnosing why a page is not indexed

Check whether the path is disallowed before investigating anything else. A blocked URL cannot be crawled, so no on-page signal will ever be read.

Verifying rules before deploying them

A trailing slash or a misplaced wildcard can block an entire section. Reviewing the parsed rule set catches it before traffic disappears.

Checking crawler-specific rules

Group precedence means the most specific matching User-agent group applies exclusively — a Googlebot group causes Google to ignore the * group entirely, which surprises people.

Confirming the sitemap reference

The Sitemap directive is independent of user-agent groups and is one of the primary ways search engines discover a sitemap.

Pro tips

Never block a page you want deindexed. Crawlers must fetch the page to see the noindex directive; blocking it guarantees the directive is never read.
robots.txt is per origin, including scheme and port. https://example.com and https://www.example.com need their own files.
Blocking CSS and JavaScript prevents rendering-based crawlers from seeing your page as users do, which harms evaluation of mobile usability and layout.
Keep the file small and simple. Comments are fine, but complex wildcard rules are the most common source of accidental site-wide blocks.

Limitations and edge cases

What this tool deliberately does not do, and where it will disagree with other implementations.

Compliance is voluntary. Well-behaved crawlers honour it; scrapers and malicious bots ignore it entirely.
It is not an access control — a disallowed path is still publicly reachable and the file itself advertises the paths you would rather nobody visited.
Non-standard directives such as Crawl-delay are ignored by Google, though some other crawlers honour them.
Rules are evaluated per origin; subdomains need their own file and inherit nothing.

Frequently asked questions

Does robots.txt keep a page out of Google?: No. It prevents crawling, not indexing. A blocked URL that is linked from elsewhere can still be indexed without a snippet. Use a noindex meta tag or header on a crawlable page instead.
Is robots.txt a security measure?: The opposite. The file is public and effectively lists paths you consider sensitive. Protect private content with authentication, never with a Disallow rule.
Which rule wins when Allow and Disallow both match?: The most specific rule, measured by matched path length. If they are equally specific, Allow wins — per RFC 9309 and Google's implementation.
Do I need a separate file for each subdomain?: Yes. robots.txt applies to one origin, so blog.example.com and example.com each need their own file at their own root.