Editorial starter: a general inspection checklist. Check each crawler’s current official documentation before changing access policies.
Inspect the response
Start with the HTTP response for an important page. Confirm the status, content type, and whether meaningful text is present in the HTML. A successful status code is only one part of accessibility.
Review crawl policies
Read robots.txt and page-level directives. Identify which user agents your policies address. Crawling, indexing, and model training are different uses; document the intended policy before changing rules.
Make content understandable
Use descriptive headings, ordinary links, and clear attribution. Ensure the canonical URL represents the intended page. Keep important content available without requiring an interaction.
Verify after changes
Inspect the built page and server response again. Monitor logs if they are available, while remembering that a user-agent string alone does not establish a crawler’s identity.