Find out if a URL is actually blocked
Longest-match precedence is why robots.txt surprises people: a broad Disallow can be overridden by a narrower Allow. This shows the winning rule, not just the verdict.
robots.txt doesn’t read top to bottom
Longest match wins
Disallow: /blog/ and Allow: /blog/public/ both match /blog/public/post. The longer rule decides, so the page is allowed — the opposite of what reading top to bottom suggests.
Named blocks override the wildcard
If any block names your crawler, the User-agent: * block is ignored for it entirely — its rules don't merge. This shows which block actually applied.
Blocked isn't de-indexed
A disallowed URL Google already indexed can stay in results, shown without a description, because Google can no longer crawl it to see the noindex tag you added.
What this doesn’t tell you: A robots.txt Allow doesn't guarantee indexing, and a Disallow doesn't remove a page already indexed — only a noindex tag or a removal request does that.
Run all of this on your whole site, continuously
These tools check one thing once. SEO Engine watches every page, every week, and writes the fixes for you.