Debugging

For: the moment something is broken and the cause is not obvious — a red gate, a failing suite, a report from a player, a deploy that did not take.

Start by asking which layer

Four layers can produce the same symptom, and the first job is to say which one you are in. Guessing wrong costs more than checking.

Layer	You are here when	First move
Your machine	It fails locally and nowhere else.	Check the prerequisites on Run it locally.
The repository's code	It fails the same way everywhere, including a fresh checkout.	Reproduce with a test, then fix.
The gate's infrastructure	It fails in continuous integration and passes locally on the same commit.	Read the container's own numbers before reading the code.
The deployment	It works locally and misbehaves only once deployed.	Compare the environment's configuration, not its code.

Reproduce before you fix

The house rule is that a claim is proved by a spec the gate re-runs. A bug that has no failing test is a bug you cannot prove you fixed.

Write the failing test first, watch it fail for the reason you expect, then fix it. A test that passes the moment you write it was testing the wrong thing.

npm test                 # the node suite
npm run test:ui          # the jsdom and React suite
npm run test:browser     # Playwright against a real Chromium
npm run test:devloop     # the dev loop's own behaviour
npm run check            # everything the gate runs, in the gate's order

Run the narrowest one that reproduces the failure. Running the whole gate to chase one broken assertion wastes minutes per iteration.

A red gate that is green locally

This has a specific and non-obvious cause.

The runner pool is not uniform. A job container's visible CPU count and its memory limit differ between runners, and node --test forks one child per visible CPU. A container with many CPUs and a small memory limit therefore starves. The memory cgroup pins at its limit, and the kernel's out-of-memory killer takes test children one at a time.

That failure does not look like what it is. The killed child emits zero subtests, and the duration printed against it is the child's own lifetime while starving rather than time spent testing. It reads exactly like one slow test. The set of victims changes between runs of an identical tree, because selection is the kernel's per-process score at the instant the limit is hit. The victim list therefore carries no information about your code.

The gate records the container's own numbers whenever it fails. Read those first:

Line in the report	What it tells you
nproc and availableParallelism	how wide the test fan-out went
memory.max	the limit the container was given
memory.peak	how close the run came to it
memory.events → oom_kill	a non-zero count proves the kernel killed children

A non-zero oom_kill means the failure is the container, not the commit. Nothing in the code will explain it, and reading the diff is wasted effort.

Two defences are in place. test:ui bounds its fan-out with --test-concurrency=4, which travels with the repository and protects any runner. The runner's own job-container memory limit is set above the default, which covers the suites that fan-out bounding cannot help. test:browser runs Playwright, Chromium, vite and wrangler dev in one container, and no test flag reduces that. That limit lives in the runner's own configuration rather than in this repository, so it is not carried by a checkout. Raising it is the remedy for a non-zero oom_kill, and it has to stay above the default for the browser suite to fit.

Reading a failed continuous integration run

The Forge's API serves no job logs, so a red run is unreadable from the API alone. The gate compensates by posting the failure back onto the pull request as a comment, in two parts.

The digest carries every line matching a failure marker with six lines of context around it. The tail carries the last 120 lines of the job. Read the digest first. A green phase can print thousands of passing lines after the phase that actually died, so a plain tail often shows a clean summary and hides the failure entirely.

When a pull request's green tick is lying

A pull request's check result belongs to the commit it ran against, not to the branch. When the head branch moves after the checks pass, the tick stays green and now refers to a commit that is no longer the head. The merge takes the branch, not the tested commit.

Before merging anything, confirm the green check's commit matches the current head. Where they differ, push an empty commit or re-trigger the checks and wait.

Where the logs are

Runtime logs for a deployed environment come from Cloudflare rather than from this repository. Cloudflare names the exact route to them, and Data and stores covers reading the state a log line refers to.

Known failure signatures

Symptom	Cause	Fix
A suite reports a long duration and no subtests.	The test child was killed for memory, not slow.	Read memory.events; bound the fan-out.
Browser specs fail with connection refused, in alphabetical order from some point onward.	A server the specs depend on died mid-run, and every later spec inherited it.	Find the first failure, not the loudest; the ones after it are consequences.
A commit reaches a protected branch with no check having covered it.	Checks trigger on pull requests, so a direct push gets none.	Land through a pull request.
A branch protection rule appears to be enforced and is not.	The required status context is one no workflow produces, so it can never arrive.	Require a context that a workflow actually emits.