How do I fix robots.txt blocking Google?
How to read the file correctly, what the precedence rules actually are, and the three mistakes that cause almost every accidental block.
Fetch yoursite.com/robots.txt and look for Disallow: /. Remember that the most specific matching user-agent group wins and that group is used exclusively — rules in other groups, including the catch-all, do not merge into it.
Read the file the way a crawler does
A crawler picks exactly one group: the one whose User-agent most specifically matches its own name. It then obeys only that group and ignores every other, including User-agent: *.
This single rule explains most confusion. In this file:
User-agent: *
Disallow: /admin/
Disallow: /tmp/
User-agent: Googlebot
Allow: /Googlebot may crawl /admin/, because it uses only its own group and that group does not disallow anything. If you want Googlebot restricted too, repeat the rules inside its group.
The three mistakes
| Mistake | What it does |
|---|---|
Disallow: / left from staging | Blocks the entire site. The most common cause by a wide margin. |
| A named group that omits rules you wanted | The named group replaces the catch-all entirely, silently dropping your protections. |
| Blocking CSS or JavaScript directories | Google renders pages; blocked assets mean it sees a broken page and may judge it accordingly. |
Fix it
- Fetch the live file:
curl -s https://yoursite.com/robots.txt. Do not read the copy in your repository — read what is served. - If you see
Disallow: /under a group that matters, remove that line or replace it withAllow: /. - Check that every group contains all the rules you intend for that crawler, since groups do not inherit.
- Confirm CSS and JavaScript paths are crawlable.
- Test the specific URL in Search Console's URL inspection, which reports whether robots.txt permits it.
- Recrawling takes hours to days. It is not instant.
A reasonable default
User-agent: *
Allow: /
Disallow: /admin/
Disallow: /cart/
Disallow: /*?session=
Sitemap: https://example.com/sitemap.xmlnoindex — a page blocked in robots.txt can never be seen to carry a noindex, so blocking it actually prevents the removal you wanted.When the file is not yours
Some hosts, CDNs and platform providers serve their own robots.txt at the edge regardless of what you uploaded. If the served file does not match your repository, that is the problem, and it is fixed in the platform's settings rather than in your codebase. Always diff the served file against the one you wrote.
What our audit reports about this
Every item below is measured directly, not inferred. Run it against your own site and the result names the exact rule or header responsible.
- The full robots.txt as served, with the rules that apply to each named crawler.
- Whether the audited URL is allowed or blocked, naming the exact rule and group responsible.
- Whether CSS and JavaScript assets are reachable.
- Whether the sitemap is referenced and whether robots.txt contradicts it.
For agents and scripts, the same measurement is at
/api/v1/discoverability?url=yoursite.com —
see the API documentation.
Related questions
How long until Google notices my fix?
robots.txt is typically refetched within a day. Recrawling and reindexing the affected pages takes longer — days to weeks depending on the site. Requesting indexing in Search Console for key URLs speeds it up.
Should I block crawlers from admin pages?
You can, but understand what it achieves: it stops well-behaved crawlers from requesting the path, and it publishes the path's existence to anyone reading the file. Real protection is authentication. Use robots.txt to save crawl budget, never as a security control.
Is Allow: / needed if there is no Disallow?
No. Everything is permitted by default; an empty group allows everything. An explicit Allow: / is harmless and is useful for readability when you want the intent to be obvious.
Read next
Why is my website not showing on Google?
Work through it in order — not indexed, indexed but not ranking, or ranking but not for anything you noticed.
ReadDo I need a sitemap.xml?
Not strictly. It helps discovery on larger or poorly-linked sites, and it costs nothing — but it will not fix a site that cannot be crawled.
ReadShould I block AI crawlers from my site?
An honest look at what you gain and what you give up, and why the answer is usually "block some, allow others".
ReadHow do I check if AI crawlers can read my site?
Four tests you can run yourself in about ten minutes, in the order that finds the problem fastest.
Read