Crawl silently loses in-Scope Pages to scheme-flipping redirect loops

[jonbakerfish]

Problem Statement

Crawling a docs site can silently return far fewer Pages than the site actually
contains, with no errors reported. Concretely: getdocs crawl https://docs.galaxea-dynamics.com/R1Lite/docs/2025/ produces 3 Pages when the
sidebar exposes 13. The Manifest shows errors: [], skipped: [],
truncated: false, and the run exits 0 — so from the user's perspective the Crawl
"succeeded" while quietly dropping 10 in-Scope Pages. There is no signal that
anything went wrong.

The cause is a redirect loop on the target site. Ten of the sidebar links are
authored without a trailing slash, and the server redirects them through a
scheme flip:

https://…/r1lite_openbox      301 →  http://…/r1lite_openbox/    (downgrades HTTPS→HTTP, adds slash)
http://…/r1lite_openbox/      308 →  https://…/r1lite_openbox    (restores HTTPS, strips slash)  ← back to start

Scrapy follows https-noslash → http-slash → https-noslash, at which point the
original request URL is already in its dup-filter, so the request is dropped
(dupefilter/filtered: 10) — no error surfaces. The 3 Pages that survive are the
Seed plus the only two sidebar links the site authored with a trailing slash
(…/r1lite_driver/, …/r1lite_motion_control/), whose HTTPS+slash form returns
200 directly with no loop.

The site's redirect config is genuinely broken (a 301 must never downgrade HTTPS
to HTTP), but getdocs should be robust to it rather than silently losing Pages.

Solution

getdocs pins a Crawl to the scheme it reached each Seed host on. When a discovered
URL or a redirect target for that same host arrives on a different scheme, getdocs
rewrites it back to the Seed host's scheme before fetching. This breaks the loop:

https://…/r1lite_openbox      301 →  http://…/r1lite_openbox/    (site downgrades)
                              pin →  https://…/r1lite_openbox/   (getdocs restores HTTPS)
                                     200 ✓   loop broken, Page fetched

With the fix, the same Crawl yields all 13 Pages. The behavior is invisible on
well-behaved sites (their redirects never flip scheme for their own host) and only
engages on the misconfiguration that would otherwise lose Pages.

User Stories

1. As a docs consumer, I want a Crawl of a site with scheme-flipping redirects to fetch every in-Scope Page, so that I get the complete documentation rather than a silent subset.
2. As a docs consumer, I want the r1lite_openbox, quick_start, controller, demo, get_sdk, ros2_com, launch_ex, teleop, and FAQ Pages of the Galaxea R1 Lite docs to be crawled, so that my output matches the site's sidebar.
3. As a docs consumer, I want getdocs to reach a Page whose only working URL differs from the authored link only by scheme and trailing slash, so that inconsistent link authoring on the source site does not cost me Pages.
4. As a docs consumer, I want a Seed I gave over HTTPS to keep the whole Crawl on HTTPS, so that a downgrade redirect cannot bounce me onto an unreachable or looping HTTP path.
5. As a docs consumer, I want a Crawl that hits a redirect loop to still terminate and produce a Manifest, so that a broken source site cannot hang or crash my run.
6. As a docs consumer, I want Pages lost to redirect loops to either be recovered or, if truly unreachable, reported in the Manifest, so that a Crawl never claims success while silently dropping in-Scope Pages.
7. As a docs consumer crawling a normal, correctly-configured site, I want the scheme-pinning to have no observable effect, so that the fix never changes which Pages I get on healthy sites.
8. As a docs consumer with multiple Seeds on different hosts, I want each host pinned to the scheme its own Seed used, so that pinning is per-host and never forces one host's scheme onto another.
9. As a docs consumer, I want the Reading Order and Nav Order in the Manifest to reflect the full set of recovered Pages, so that prev/next chains and the sidebar tree are complete.
10. As a docs consumer, I want a resumed Crawl (--resume) to apply the same scheme-pinning, so that Pages left pending by a loop in an earlier run are recovered on resume.
11. As a maintainer, I want the redirect-loop fix covered by an e2e test that drives a scheme-flipping redirect through the real CLI, so that the regression cannot silently return.
12. As a maintainer, I want the fix implemented as a downloader middleware alongside RetryAfterMiddleware, so that it composes with Scrapy's own RedirectMiddleware and follows the established pattern in the engine.
13. As a maintainer, I want scheme-pinning to leave cross-host redirects untouched, so that a legitimate redirect to a different host on its own scheme still works.
14. As a maintainer, I want the Scope check to continue to gate the final (post-pin) URL, so that scheme rewriting can never smuggle an out-of-Scope Page into the Crawl.
15. As a docs consumer, I want Asset URLs (which are exempt from Scope per ADR-0005) to be unaffected by scheme-pinning of Pages, so that off-host media still downloads normally.

Implementation Decisions

• New downloader middleware in engine.py (working name: scheme-pin / SchemeLockMiddleware), registered in DOWNLOADER_MIDDLEWARES next to RetryAfterMiddleware. It is the single seam for the fix. The spider modules (scope, urlnorm, navharvest, etc.) are unchanged.
• Scheme map is per-host, derived from the Seeds. getdocs records, for each Seed's host, the scheme of that Seed URL. The middleware consults this map; hosts with no Seed are not pinned.
• The middleware rewrites requests, not Location headers. It acts on outgoing requests (including the redirected requests that Scrapy's RedirectMiddleware re-injects into the chain). For a request whose host is in the scheme map and whose scheme differs from the pinned scheme, it replaces the request with an otherwise-identical one on the pinned scheme. This is order-robust: it does not depend on running before/after RedirectMiddleware, because the redirected request re-enters the downloader middleware chain and is corrected on its next pass.
• Only the scheme is changed. Host, port, path (including any trailing slash the redirect added), query, and fragment are preserved. This is what turns http://…/openbox/ into https://…/openbox/ — the working 200 form.
• Cross-host redirects are untouched. If a redirect points at a host not in the scheme map, the middleware leaves it alone.
• Scope still gates the post-rewrite URL. Rewriting scheme never bypasses Scope.allows; the existing check already runs on the final URL. Scope is host + path based, so the scheme rewrite does not change Scope membership.
• Frontier dedup is unaffected. urlnorm.normalize already lower-cases and normalizes scheme/host and strips trailing slashes; two forms of the same Page continue to dedup to one entry, so the fix does not create duplicate Pages.
• Interaction with the loop: with the pin in place, the http://…/x/ hop is rewritten to https://…/x/, which the server answers 200 — so the loop terminates at a real Page instead of bouncing back to a dup-filtered URL.
• Resume compatibility: the pending frontier persisted in the resume state is re-yielded on --resume and passes through the same middleware, so previously-lost Pages are recovered without special handling.

Testing Decisions

• What makes a good test here: it drives the real getdocs crawl CLI against a fixture site and asserts on observable Crawl outputs — which .md Pages were written and the Manifest counts — never on the middleware's internals. This mirrors how RetryAfterMiddleware is verified (through a full Crawl, asserting the recovered Page exists and timing is honored) rather than by unit-testing the middleware object.
• Seam: the single seam is the full-crawl e2e path via the shared FixtureSite (site fixture) plus run_getdocs(...). Prior art: tests/test_traversal_e2e.py (redirect following, dedup, Scope) and tests/test_politeness_e2e.py (middleware behavior observed through a Crawl).
• Reproducing the loop in the HTTP-only fixture: the fixture serves one host over HTTP. Because the fix pins to the Seed's scheme (HTTP in the fixture), the test seeds over HTTP and adds a route whose redirect Location flips to https://<same-host>/… (a scheme the fixture cannot serve). Without the fix the redirected request goes to an unreachable/looping scheme and the Page is lost; with the fix it is pinned back to HTTP and fetched. This faithfully exercises the same mechanism as the real HTTPS→HTTP case, just with the direction that the fixture can serve.
• Assertions: the looped Page's .md is present, page_count/md_files include it, and the Crawl exits 0. A companion assertion confirms a cross-host or same-scheme redirect still behaves as before (no over-reach).
• Module under test: the engine/crawl path (via CLI). New fixture helper may be added to conftest.py only if add_redirect cannot already express an absolute scheme-flipping Location (it takes a location string, so it likely can as-is).

Out of Scope

• Fixing the target site's redirect configuration — that is the site owner's problem; getdocs only makes itself robust to it.
• A general redirect-loop detector or max-redirect surfacing for arbitrary loops not caused by scheme flips. (Scrapy's own redirect limit still applies.)
• Honoring rel=canonical to pick a URL form — explicitly rejected by ADR-0003; canonical is recorded, not followed.
• Normalizing trailing slashes at enqueue time as a separate strategy; the scheme-pin already resolves the observed failure, and slash normalization risks conflating genuinely-distinct URLs.
• Adding HTTPS/TLS support to FixtureSite.
• A user-facing flag to disable scheme-pinning; it is on by default and inert on healthy sites. Can be revisited if a real HTTP-only site regresses.

Further Notes

• During diagnosis, WebFetch masked the bug because it auto-upgrades HTTP→HTTPS; the site "looked fine" in that tool while Scrapy faithfully followed the literal http:// redirect into the loop.
• Evidence from a fresh reproduction: Scrapy stats reported dupefilter/filtered: 10, finish reason finished, errors: []; the resume-state pending map held exactly the 10 lost URLs. curl confirmed the three-way status chain (301 → http+slash, 308 → https+noslash, and https+slash → 200).
• The two Pages that succeed today do so only by the accident of having trailing slashes in their authored sidebar hrefs.