Checks if a robots.txt file exists and parses crawl directives.
Want all 12 checks at once? Run a full site scan →
