Robots Policy Auditor
Evaluate a local robots.txt, captured X-Robots-Tag headers and captured HTML robots directives for selected agents and URLs. Review which rule won before a release without fetching a site.
v0.1.0 · Node.js 22+ · MIT
Browse source and run locally · Technical SEO release checklist
Compare two public policy fixtures
Clone the repository, use Node.js 22 or newer, and run these commands from its root. The inputs are synthetic and checked in.
Expected crawl and indexing decisions
node bin/robots-policy-auditor.mjs --config examples/clean/robots-audit.config.json --jsonThe clean fixture completes 7 checks with 0 errors and 0 warnings. It reports 4 crawl-allowed and 3 crawl-disallowed decisions; one captured indexing directive blocks indexing for its checked response.
Contradictory or ineffective directives
node bin/robots-policy-auditor.mjs --config examples/broken/robots-audit.config.json --jsonThe deliberately broken fixture checks 5 cases and reports 4 errors and 8 warnings. Among them, it flags a noindex directive behind a Disallow, where the compliant crawler cannot fetch the response to read that directive.
What these decisions can and cannot establish
The CLI reads supplied local files and captured response evidence. It does not fetch pages or prove indexing, ranking, crawler compliance or the current state of a live site. Crawl permission and indexing directives are separate decisions: a Disallow is not de-indexing, and private content needs access control rather than robots rules. Missing capture evidence is reported as unverified, never as a pass.
Continue the release review
Check crawl rules before they affect a release
Discuss a technical SEO release check with Edilec. Share only site details you are permitted to disclose.