Skip to content

Flaky a11y audit test times out + cascades in the pre-push gate #108

Description

@thalida

tests/a11y/audit.test.tsx (the axe-core audit added in #79) intermittently fails the pre-push gate (and would in CI), while passing reliably locally in ~6s. It bounced a README-only push twice in one session before a retry went green.

Symptoms

  • One or more <surface>: no axe violations cases fail, often several at once (a cascade, not a single flake).
  • Local npx vitest run tests/a11y/audit.test.tsx → 10/10 pass, ~6s.

Root cause

Two things compound:

  1. Timeout under load. axe.run on the large settings/ControlsPane DOM occasionally exceeds the per-test 30_000ms timeout when the docker gate is running pytest + vitest + coverage under resource contention (CI/gate is much slower than a dev box). The test already trims cost (scans only BUILDINGS_SECTION via axeMount, resultTypes: ['violations'], contrast/region rules off) — but it's still on the edge.
  2. axe is a singleton → timeouts cascade. The test's own comment notes it: a run that times out mid-flight leaves axe "running," so the next surface throws. So one slow surface fails multiple its and bounces the whole push.

Directions (pick some)

  • Break the cascade (highest value): ensure axe is reset between surfaces even after a timeout, e.g. afterEach calling axe.teardown() / axe.reset(), or serialize surfaces so one wedge doesn't poison the rest. This alone turns a multi-failure bounce into at most one flaky case.
  • Cut the wall-clock dependence: further shrink what's scanned per surface, or split the audit so each surface is its own isolated run.
  • Reconsider gate placement: keep the cheap structural guards in the pre-push gate, but move the heavy axe.run portion to CI-only (jsdom axe is slow and gate contention makes it flaky) — the structural guards catch most regressions fast.
  • As a floor, bump the timeout, but that treats the symptom, not the singleton cascade.

Impact

Low correctness risk (it's a test, and it passes locally), but it costs real retries and erodes trust in the gate. Related: #79 (where the audit was added).

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions