robots.txt is blocking your whole site
robots.txt is advice to search engines. User-agent: * followed by Disallow: / tells every crawler to skip everything. It is usually left over from a staging site.
- Difficulty
- easy
- Time
- about 10 minutes
- Last reviewed
- 2026-10-26
Is this urgent?
Today.
Easy fix, no technical skills
Open yoursite.com/robots.txt in a browser and look for Disallow: / under User-agent: *.
WordPress: if a SEO plugin manages robots.txt, edit it there (Yoast: Tools, File editor; Rank Math: General Settings, Edit robots.txt). Otherwise edit the file in your host's file manager.
Replace the blocking rules with: User-agent: * then Disallow: /wp-admin/ then Allow: /wp-admin/admin-ajax.php then Sitemap: https://yoursite.com/sitemap.xml
Hosted platforms generate robots.txt for you; check their SEO settings for a hide-from-search option instead.
Hand it to your IT person
Copy this into an email or chat. Replace the brackets with the details from your alert.
robots.txt at example.com disallows / for all agents. Please replace it with the standard rules for the platform and resubmit the sitemap in Search Console.
Technical fix (commands)
Read it.
curl -s https://example.com/robots.txt
Check it worked
Search Console robots.txt report shows no errors; our scan clears the finding.
Scan the site's robots.txt with the free tool. All free tools.
Prevent it next time
- Never copy a staging robots.txt to production.
Common questions
Is this urgent?
Today.
Can I fix this without technical skills?
Yes. The easy fix below needs no technical skills; the commands are only for your IT person.
How long does it take?
About 10 minutes once you know who runs the website.
Hear about this before your customers do
Add your website and we check uptime, certificates, domains and email security for you, free.