Search

Disallow before User-agent: the robots.txt mistake that blocks nothing

Disallow before User-agent: the robots.txt mistake that blocks nothing

A file that starts with Disallow: /admin and never names a user-agent is not a strict robots.txt. Crawlers look for a User-agent: group first. Directives above that line are easy to ignore. The same file with User-agent: * then Disallow: /admin is the policy you thought you wrote.

Also Read: Why Slack and iMessage show Content the wrong title for your post

Groups, Allow, and Sitemap

You can have more than one group. A Googlebot group with Allow: / and a * group that disallows /admin is a normal pattern. Sitemap: lines should be absolute https:// URLs, not /sitemap.xml. Crawl-delay is numeric when you use it at all; many crawlers ignore it.

Also Read: SEO: UTM parameters that

The robots.txt and sitemap validator checks that order, unknown directives, sitemap loc / lastmod, and optional public fetches (http(s) only, no localhost, no private IPs). Paste first. Fetch if you want to see what production actually serves.

Also Read: Moving Apache Redirect and RewriteRule lines to nginx

Cutover and www

If you just moved hosts, the validator will not fix nginx for you. Redirects belong in the htaccess to nginx pass. Once the origin is stable, keep og: tags on the canonical host so social unfurls and crawlers agree; that is the OG preview problem, not robots.

Re-fetch after deploy. A CDN can serve a week-old robots.txt while your origin is already correct.

Tags:
TWT Staff

TWT Staff

Writes about Programming, tech news, discuss programming topics for web developers (and Web designers), and talks about SEO tools and techniques

Your experience on this site will be improved by allowing cookies Cookie Policy