AI in QA isn't about writing tests

I replaced waitForLoadState('networkidle') (discouraged) with a Svelte $effect on the root layout that sets body[data-hydrated="true"] after mount. Tests fence clicks behind the marker: fast, deterministic, and an actual hydration signal rather than a network-quiet heuristic.

Children mount before parents in Svelte, so by the time the root layout's effect fires, every SongRow / modal / form on the page has its onclick bound.

SvelteKit use:enhance re-runs load() in place without remounting, so the hydration marker stays "true" from the previous render. Form-action submits need a distinct fence: waitForResponse on the action's POST.

I documented the distinction inline in helpers/hydration.ts so the next reader doesn't regress it: hard navigations get awaitHydrated, form actions get waitForResponse.

I instructed AI to use ARIA roles (getByRole), accessible labels (getByLabel), and data-testid over CSS classes or hard-coded ids. In doing this, tests target the accessibility surface, not the markup.

Concrete payoff: 10/10 group specs survived four extractions of song-row markup unchanged.

I scoped every seeded identity to its Playwright worker: profile handles get a worker suffix (sam-w0), group slugs get a worker prefix (e2e-w{N}-...), and seeded emails are unique per test, so concurrent workers never collide on unique constraints. wipeTestData matches the same patterns and deletes only its own worker's rows.

With 5 workers in parallel we now have zero cross-talk. Worker 0's sam-w0 never sees worker 3's sam-w3.

When the third-party resolver wasn't the thing under test, markAllSongsResolved faked the Odesli-resolved state directly via the service-role Supabase client. That turned a 14-line "wait for external" workaround into a 2-line setup.

The lesson I took from it: stub at the data layer when you're not testing the integration. The browser doesn't need to know.

creator.mixtape.addSongsByList(...). group.editName(...). group.shareMyMixtape().

Tests read as sentences in the product's domain, not DOM walks. The mechanical "which selector targets which button" detail stays inside the page object, where it can change once behind a stable method name.

Spec 12 (first-visit hint) had a 14% flake rate invisible when run a single time. The hint dismissal depended on a row click; under load that click occasionally landed before SvelteKit's hydration bound the handler.

At 100× the pattern was obvious: same spec, same line, every time. Without stress, it would have shipped.

"Click timed out" was the symptom that I was seeing. "Click landed before SvelteKit bound onclick" was the actual cause. The naive fix, bumping the timeout, would have masked the issue forever.

The real fix was framework-wide, not per-spec: a single hydration marker replacing eight ad-hoc networkidle calls.

Adding the masthead inline-edit E2E spec exposed requireMixtapeOwner's .is('group_id', null) filter not matching the seeded data, the same shape as a bug the load function had already absorbed.

The test framework acted as a forcing function. The shipped code "worked" only because nobody had exercised the gate from a fresh client.

An AI audit agent reported the reserved-handle denylist as a "real product hole." I pushed back; I knew one existed. Verifying directly found it inline in a route file, against the project's documented convention.

The lesson: AI audits surface candidates, not conclusions. Verify before alarming. When an audit says "X is missing," check yourself before passing the alarm along.

I ran three agents in one batch: an accessibility audit, a duplication and cohesion audit, and a coverage-gap audit. Each got a tightly-scoped self-contained brief and reported back a punch list grouped by severity.

Total wall-clock was under 4 minutes for a sweep across the whole codebase. I reviewed all three lists and picked what to act on.

We added aria-label to seven placeholder-only inputs and role="alert" to form errors, verified the accessible-name compute for the masthead pencil and heading, and reviewed <svelte:body> for landmark roles.

All of it was treated as part of the work, not a separate "accessibility pass" that gets cut when timelines slip.

The reserved-handle predicate got a fast Vitest (8 assertions, 109ms). The masthead inline edit got an E2E spec because the journey crosses InlineEdit + page state + server action + RLS.

Pick the right tool per layer. Not religion about either.

When the suite-wide networkidle cleanup was flagged but not yet warranted, it went into a single docs/TODO.md entry, with the "right replacement to pick" question called out.

When I later came back to act on it, the implementation choice was pre-baked. No re-litigation, no re-diagnosis.

I classified every red test report into one of three buckets before attempting any "fix":

• Product bug — the app is wrong. Fix in app code. Example: the requireMixtapeOwner filter that didn't match seeded data.
• Test gap — the test is wrong (bad selector, missing fence, wrong assertion shape). Fix in test code. Example: the hydration race, asserting response.status() === 303 on a SvelteKit form action that returns 200.
• Local infra flake — Supabase returned "invalid response from upstream server," Mailpit hadn't delivered yet, a waitForURL tail past 30s. Accept, retry, move on.

Concretely: in the final 100× run, two specs flaked on waitForURL, both classified as infrastructure tail within minutes (Supabase / Mailpit slowness, recovered on retry, no commit needed). In the same session, the same triage flagged the requireMixtapeOwner failure as a real product bug, even though it surfaced through a test setup error message.

The classification short-circuits the common anti-pattern of "test failed → mute the test." Knowing which bucket a failure belongs in determines whether the fix touches app code, test code, or neither. (The infra bucket later got a mechanical implementation instead of a human shrug; see card 24.)

I ran the suite 100× before the fix, 100× after the fix, and 100× again after adding spec 14. Each commit was grounded in an empirical pass-rate, not "it works on my machine."

The test framework was both the safety net and the verification tool for changes to itself.

The page-object layer was built for tests. It turns out the same vocabulary doubles as a video script: creator.mixtape.addSongsByList(...) is simultaneously a test action and a narratable step.

A separate playwright.demo.config.ts runs the demo flows serially with video recording on. Same fixtures with one twist: seedUser grew a literalHandle mode so URLs in the recording read cleanly (/diane, not /diane-w0). A dedicated wipeDemoData handles the fixed-handle cleanup that worker-scoped tests don't need.

The flagship recording is a side-by-side composite of two writing-group members (Jack and Diane) building their mixtapes in parallel, which works because the framework already orchestrated multi-actor journeys (spec 08 runs invite + join across two browser contexts). The demo capability fell out of primitives already paid for.

Why this matters for QA-as-a-business-asset: every page object the team writes pays back twice. Once as a regression net, once as onboarding / sales / internal-training content. No separate content pipeline, no separate vocabulary, no drift between "what the tests assert" and "what the demo shows."

The local default is the fast line reporter. CI=1 switches to dual list + html with the report written to a known path. One flag, two modes, no per-run config edits.

The friction to "show me what failed" stays low whether running locally or in CI.

The day Claude Fable 5 shipped, I pointed a fresh Fable 5 window at the codebase the original session had built and asked it to review. It caught real things the working session had missed: a banned vocabulary word ("track") in a modal we had read minutes earlier, a two-line empty state that violated the project's "one line each" rule, design-language drift on the edit page. Every one was a rule compliance failure: rules the working session knew and had stopped applying.

But the same audit also produced confident mischaracterizations. The one flagged as "fix this first," a claimed cross-mixtape data leak in the OG endpoint, turned out to be half-right: the reader page already had the mixtape_id scoping with a comment explaining why, but the OG endpoint genuinely still queried by owner_id and was fixed that session. Right file, real inconsistency, wrong severity: current share semantics guarantee one mixtape per user, so it was defense-in-depth, not an active leak. Two UI findings (a ViewToggle a11y "issue" on buttons that had visible text labels, a "hardcoded handle" on the landing page's own owner) were wrong or intentional, and a flagged admin-endpoint "auth gap" was the endpoint working as designed. Scorecard: ~60% genuine catches, ~40% false positives or mischaracterized severity. The audit isn't a fix list; it's a candidate list. Verification is non-optional.

The deeper observation is that the dominant variable wasn't "newer model." It was fresh context. No mid-session bias, no "I just read this file, surely it's fine," no half-remembered rules from earlier turns. A fresh session of the same model, prompted as an auditor rather than a builder, would catch most of the same things. The capability uplift helped on the genuinely cross-file findings (the post-groups data-model concerns); the rule-compliance catches were pure attentiveness.

The practical move: you don't need to wait for the next model release. After any sizable change (or quarterly, or after onboarding a new contributor) open a fresh session, paste the rules (CLAUDE.md, design-language.md, whatever you have), and ask for a compliance walk-through. Treat the output as a candidate list. Verify each finding. Act on the real ones. Like a colleague coming back from vacation: the room hasn't changed, but they can see it again.

A ×100 stress run produced one failure that looked like silent data loss: a story saved "successfully," then wasn't on the public page. The save's POST had returned HTTP 200, but SvelteKit's use:enhance answers 200 even when the action returned fail(...). The failure lives in the body: {"type":"failure","status":500,"data":"...invalid response was received from the upstream server"}. The local Supabase gateway had choked under 14 parallel workers; the product handled it correctly and surfaced the error; the test harness waited for a response without reading it, sailed past, and died two steps later on an unrelated-looking assertion.

The fix is the sequel to card 02: the form-action fence was necessary but not sufficient. A shared actAndExpectSuccess() helper now parses the ActionResult body at all five wait sites and throws the action's own error payload at the failing step. Next time the failure message is the diagnosis.

Generalizable far beyond SvelteKit: any framework that tunnels application errors inside transport-level success (GraphQL's errors array inside HTTP 200, gRPC status in trailers, batch APIs with per-item failures) has this trap. "Waited for the response" and "checked the response" are different fences.

The card-20 diagnosis never re-ran the test. Playwright's retain-on-failure trace held the whole story: the POST body (proving the full story text was submitted), the response body (proving the action returned failure-inside-200), and millisecond-level request ordering (proving the page load really did start 6ms after the save response, ruling out the obvious race). Unzip the trace, read the .network entries, done.

The last link came from an accident worth keeping: per-test namespaced data (sam-w364) never gets wiped, so the exact rows from the failed run were still in local Postgres an hour later: story row present, text empty, timestamp matching the import, confirming the upsert never landed. The "leftover e2e data" on the cleanup TODO turned out to be a forensic archive.

The discipline that made it efficient (learned the hard way, mid-session): after a long run, report the pass/fail summary and failure signatures first; then state a one-line hypothesis before each check. Ten failures collapsed into five signatures, each got a hypothesis, and every command answered a stated question instead of being one more mystery prompt to approve.

TESTING.md's tag-enforcement section says, in writing: the check is format-only, so semantic drift like @feature:groups would slip through: "if we ever see that happen, layer in a strict allowlist." Spec 15 then shipped with @feature:edit next to the canonical @feature:editor. The format check passed; the drift sat there until a doc-refresh pass ran the one-line tag inventory (grep -ohE "@(feature|role):[a-z-]+" | sort -u) and the duplicate jumped out.

Two lessons. Documented known-gaps work as standing audit queries: the doc told us exactly what to look for and exactly what to do on first occurrence. And "bring this doc up to date" is a surprisingly effective QA prompt: refreshing stale documentation forces an inventory of reality, and inventories find drift.

An audit suggested adding aria-labels to the Expanded/Compact toggle buttons. It sounded plausible until a check of the page objects: getByRole('button', { name, exact: true }). An aria-label replaces a button's accessible name, so labels like "Expanded view — show every story in full" would have broken every spec that locates the toggle by name.

The right move was reverting the labels, not loosening the tests. The buttons had visible text; that text was the accessible name; the tests were enforcing exactly the contract WCAG's label-in-name rule wants. When tests target the accessibility surface (card 03), a "helpful" ARIA addition is a breaking API change, and the suite catching it is the system working, in the opposite direction from usual.

Once the harness could read ActionResult bodies (card 20), the obvious next ask was "on a 500, wait a few seconds and try again." The implementation matters more than the feature: the helper classifies the failure first. A 5xx, the gateway-under-load signature from the stress run, gets exactly one re-trigger after a 3s pause. 4xx and validation failures throw immediately, because retrying those doesn't make a bug less real, it makes it invisible.

The principle worth stating plainly: a retry is only defensible when it encodes a hypothesis about why the failure is transient. "Local gateway chokes under 14 parallel workers" is a hypothesis; "maybe it'll pass this time" is a coin flip. Global throw-it-at-the-wall-until-it-sticks retries are the worst kind of green: they launder product bugs into eventually-passing checks, which is strictly worse than flakiness because nobody investigates a pass.

The sharp edges were named in the commit rather than papered over: a 500 that lands after the DB commit can make the retry surface a duplicate-style app error ("slug taken"), and one action's UI state isn't re-triggerable after failure. Both fail loudly at the right step. Known-and-documented beats handled-and-hidden.

A launch-night bug report: "paragraph breaks on stories disappear — wall of text." The diagnosis was one grep deep and three phases old: both story-rendering components wrapped their markdown output in class="prose-story"… and that class was defined nowhere. It had been referenced since the editor first shipped. Tailwind ignores unknown classes silently, and its preflight zeroes paragraph margins, so every multi-paragraph story on every surface had rendered flush since day one.

This is the inverse of the earlier abstraction lessons. BrandCap (card after the stress run) was an abstraction earned and then built; the aria-label revert (card 23) was an addition correctly rejected; this was an abstraction named but never built, and nothing in the toolchain says "you referenced a class that doesn't exist." Type checking catches phantom functions; nothing standard catches phantom classes. A small lint (or even a grep in CI comparing referenced custom classes against defined ones) would have flagged it at the original commit.

The sharper QA lesson is why 4,200 stress-run executions never saw it: the seed data's stories were single paragraphs. The styling gap only becomes visible when content exercises it: a real human wrote five paragraphs about Prince, and the bug appeared within hours of launch. Coverage isn't just code paths; it's content shapes. The fixture follow-up writes itself: at least one seed story with multiple paragraphs, a list, and a blockquote, so the prose path is exercised by something shaped like real writing.

The phantom-class bug (card 25) forced the question every UI team dodges: how do you write a Playwright test for "does this look right"? The answer was to refuse the question: it decomposes into three smaller ones, and only the last needs judgment.

Layer 1: did the pipeline produce the right tags? Pure function, unit test. Seven Vitest cases pin the markdown renderer's output (paragraphs, hardened links, escaped HTML, dropped images), including security behaviors that had never been pinned. Layer 2: does the CSS actually bind to those tags? This is where both real bugs lived, and Playwright answers it deterministically with toHaveCSS: no screenshots, no judgment. The reframe that makes it work: don't assert "looks right," assert the computed properties that encode a documented design decision. The design language says paragraphs get 0.75em of air and links get a 2px accent underline. Those are numbers (margin-top: 12px, text-decoration-color: rgb(176,74,47)), and numbers are testable. One spec, a fixture story containing every shape the renderer can emit, five assertions, ~2 seconds, zero flake surface. Mutation-verified: undefine the class, the spec fails; restore it, the spec passes.

Layer 3: genuine visual judgment. White text on white background passes every computed-style assertion. The honest options were both deliberately deferred, with reasons stated: toHaveScreenshot() pixel-diffs mechanize "a human judged it once" but flake across platforms until there's CI on a fixed runner; an LLM judge answers the real question but costs money per run and renders nondeterministic verdicts, so it belongs out-of-band (audit passes, or triaging a pixel-diff failure: "does this change look intentional or broken?") and never blocks a commit.

The card-worthy move isn't the two solved layers; it's solving the deterministic ones for free and naming the third as unsolved. A suite that pretends screenshots-everywhere or judge-everything would have been better is a suite nobody runs. Scope the judgment out loud.