Approval rejected5,973 67.7% of failures
What the traces show
All 5,973 occur in Cursor/Grok. The largest component is 3,394 requests without a URL, principally search requests. This is a model-runner concentration, not a pattern of websites refusing crawlers. Even public government and documentation URLs were rejected.
Website owner
No website change indicated. Your page may never have been requested.
Harness / browsing-tool owner
Configure scoped web-tool approvals; verify search and fetch with a canary before collecting answers.
Cited examples and successful counterexamples
Inspect matching attempts →
URL safety verification failed1,123 12.7% of failures
What the traces show
The pattern spans both Codex models and both Claudes, but 960 of 1,123 events come from Claude. Hugging Face and GitHub lead the named hosts. Familiar, public hosts appearing here shows why this label must not be read as a malware verdict.
Website owner
Verify public DNS, TLS and canonical redirects. Check whether a request reached your edge.
Harness / browsing-tool owner
Expose the URL-validation reason and distinguish preflight rejection from a server response.
Cited examples and successful counterexamples
Inspect matching attempts →
Other / insufficient error detail541 6.1% of failures
What the traces show
411 of 541 come from Codex, where some saved results contain only an Internal Error header. There are also generic fetch failures from SWE and Claude. The category mixes mechanisms; rankings inside it cannot identify a shared root cause.
Website owner
Correlate the exact URL and time with edge and origin logs before changing policy.
Harness / browsing-tool owner
Retain sanitized errors, response metadata and timings so a generic failure can be diagnosed.
Cited examples and successful counterexamples
Inspect matching attempts →
HTTP 403 — refused request268 3.0% of failures
What the traces show
211 of 268 occur in SWE. Semantic Scholar, Schwab Asset Management, SEC and Make recur. This crosses scholarly, financial and software sites; selective client/IP policy is a hypothesis, not a proven shared explanation.
Website owner
Review the precise deny rule if the resource should be public. Preserve intentional access controls.
Harness / browsing-tool owner
Record client identity and the response; do not infer a WAF or paywall from 403 alone.
Cited examples and successful counterexamples
Inspect matching attempts →
HTTP 404 — missing resource235 2.7% of failures
What the traces show
SWE accounts for 177 of 235; Grok accounts for 27. Cursor documentation and Zustand paths recur, alongside local-service sites. This is consistent with stale or guessed deep links, but the trace alone cannot distinguish a model-invented URL from a removed page.
Website owner
Redirect genuinely moved pages, update internal links and keep real 404s for nonexistent paths.
Harness / browsing-tool owner
Use canonical navigation or search to recover stale links; preserve the originally requested URL.
Cited examples and successful counterexamples
Inspect matching attempts →
Timeout212 2.4% of failures
What the traces show
All seven models have timeouts. Microsoft.com accounts for 42 of 212, while failures also affect investor pages, workspace pages and GitHub. Cross-model presence argues against treating every timeout as one CLI approval defect. It does not establish a site-wide outage.
Website owner
Measure response times and inspect slow origin work or redirect loops for the affected URL.
Harness / browsing-tool owner
Record the timeout threshold and elapsed time. Use bounded retries, not silent unlimited retries.
Cited examples and successful counterexamples
Inspect matching attempts →
Connection / DNS failure209 2.4% of failures
What the traces show
205 of 209 occur in Claude; the other four are Muse search transport failures. Google documentation, local real-estate sites and HumanSignal recur. That distribution may reflect transport or logging differences as well as site reliability.
Website owner
Check DNS, TLS and connection-reset logs from independent networks.
Harness / browsing-tool owner
Distinguish DNS, TLS, socket and provider transport errors; capture timing and whether any response arrived.
Cited examples and successful counterexamples
Inspect matching attempts →
Rate limiting105 1.2% of failures
What the traces show
81 of 105 occur in Muse; 84 have no named target host. Much of the signal is search-provider capacity, not a website throttling the model. Named targets include Devin, Redfin and Fiscal, which deserve separate investigation.
Website owner
For your own 429s, document limits and return useful retry guidance. A search-provider quota is outside your control.
Harness / browsing-tool owner
Reduce concurrency, respect backoff, cache repeated research and identify which service issued the limit.
Cited examples and successful counterexamples
Inspect matching attempts →
Unsupported content type93 1.1% of failures
What the traces show
All 93 report text/markdown, and all are in Codex: 76 Astra, 17 Sol. Sentry contributes 30, Neon 14 and Prisma nine. This concentration suggests reader/representation compatibility rather than a general failure to fetch documentation.
Website owner
Respect requested representations, keep HTML available and separate cache variants. Do not assume an AI user agent supports Markdown.
Harness / browsing-tool owner
Support the received format or explicitly negotiate one the reader can parse. Record request and response headers.
Cited examples and successful counterexamples
Inspect matching attempts →
HTTP 401 — authentication required7 0.1% of failures
What the traces show
All seven occur in SWE: six target Hugging Face and one api.openai.com. This is a small cluster of potentially gated resources and an API endpoint, not broad evidence that public documentation is inaccessible.
Website owner
Keep intended authentication; offer public descriptive pages and clear authorized-access instructions.
Harness / browsing-tool owner
Use public documentation as the research source. Do not treat credentialed API endpoints as ordinary pages.
Cited examples and successful counterexamples
Inspect matching attempts →
Response too large6 0.1% of failures
What the traces show
All six occur in Codex. Cohere reference pages account for three and Claude API documentation for one; two have no resolved host. The small sample points toward documentation payload limits, not a general site availability problem.
Website owner
Offer focused, bounded documentation pages and avoid unnecessarily large application payloads.
Harness / browsing-tool owner
Report the reader’s byte limit and support scoped retrieval where available.
Cited examples and successful counterexamples
Inspect matching attempts →
HTTP 500 — server error4 0.0% of failures
What the traces show
All four occur in SWE; three point to Technavio and one to an insurance site. These are repeated observations at two hosts, not four independent incidents. The origin versus intermediary responsible is not identified.
Website owner
Check application and gateway logs, dependencies and route health for the recorded time.
Harness / browsing-tool owner
Retain status and error details, distinguish intermediary errors and use conservative retries.
Cited examples and successful counterexamples
Inspect matching attempts →
Access challenge in returned content4 0.0% of failures
What the traces show
All four detections occur in SWE: three on an SF Chronicle article and one on PyPI. The reader got a response, but it asked for JavaScript or human verification. This illustrates why HTTP success and usable research content differ.
Website owner
Where automated public reading is intended, provide a supported static path or documented API.
Harness / browsing-tool owner
Detect challenge content rather than counting any returned body as a successful read. Respect access policy.
Cited examples and successful counterexamples
Inspect matching attempts →
HTTP 402 — payment-related status4 0.0% of failures
What the traces show
All four occur in SWE: three on docs.sweep.dev/pricing and one on Investopedia. Four status observations do not establish four paywalls; 402 has no standardized payment workflow and site implementations vary.
Website owner
Inspect the actual response and configuration; publish clear public pricing/access information where intended.
Harness / browsing-tool owner
Retain the response. Do not infer a standardized paywall workflow from status 402.
Cited examples and successful counterexamples
Inspect matching attempts →
HTTP 503 — unavailable service2 0.0% of failures
What the traces show
Both observations occur in SWE at prompts-gpt.com, on the homepage and pricing page. This is one-host evidence with too little coverage to infer a broader trend.
Website owner
Check maintenance, origin health and upstream availability; provide retry guidance for temporary outages.
Harness / browsing-tool owner
Use bounded retry/backoff and retain the initial failure rather than replacing it with the retry outcome.
Cited examples and successful counterexamples
Inspect matching attempts →