Preserving the Full-Run Verdict When Only a Subset of Tests Executes
Selective testing matches a full run only if evidence backs every skip.
Sponsored by Graphide

Selective testing preserves the verdict of a full CI test run only when evidence, from execution traces or a graph of the code, identifies every test a change can affect, the system runs all of them, and unknown reachability triggers a conservative fallback. Without them, skipping tests rests on an assumption about what would have passed.
Why subset testing is not the same thing as selective testing
Pressure to shorten CI eventually becomes pressure to run fewer tests. A team waiting on a slow suite might remove its longest tests from the pull request run, or suppress tests that repeatedly fail. The wait becomes shorter, but the evidence becomes weaker. Neither runtime nor failure frequency establishes that a test cannot detect a regression in the change under review.
An evidence-based system approaches the same pressure differently. It excludes tests because analysis establishes that the change cannot affect the behavior they observe.
What reachability actually means in a test selection context
A test is reachable when a change in one source location can affect the behavior it observes. File proximity is not enough to establish it. Neither is membership in the same module or ownership by the same team. A reachability map connects source locations to the tests that execute them, so a change can be matched to the tests whose results it could alter.
Stripe illustrates how that relationship can be grounded in execution. Its selective test execution system records which files each test reads while it runs, across a Ruby monorepo of about 50 million lines, and uses bitmaps to combine the tests associated with changed files. The important property is where the relationships come from. They reflect executed paths rather than an approximation based on directory layout. Other selectors build the map statically, from a graph of what calls what in the code.
That map must account for more than direct calls. Shared utilities, configuration, and runtime-loaded modules can connect a change to tests far from the edited file. A map can miss some connections by default, including platform-conditional behavior, dependencies on data or infrastructure state, and dynamic dispatch that static analysis cannot resolve. A Linux execution trace cannot, by itself, establish what a Windows- or macOS-specific path will do.
How can selective testing preserve the result of a full CI test run?
The first condition is map accuracy. The execution traces must reflect the code before the change and cover the paths that could be exercised under it. A map built from a codebase that has since diverged can silently select the wrong subset. Structural changes therefore require the map to be rebuilt or validated, including an added dependency, a restructured module, or an altered build configuration. A nightly map is not current merely because its collection job is called nightly.
The second condition is selection completeness. Every test identified as reachable must run, without further exclusions for flakiness, runtime, or cost. Otherwise, the system discards its own evidence. A reachable test that is expensive to execute remains capable of changing the verdict, and suppressing it creates a gap. Flaky tests need to be handled on their own because hiding them inside selection hides both test instability and whether selection is correct.
The third condition is map completeness over the change. Every changed location needs coverage sufficient to support a reachability determination. If no test has ever executed a changed path, the absence of a connection in the map does not prove that no connection exists. The system must flag unknown coverage and run the full suite or the most conservative available superset.
A test whose result could change is selected, while new and previously failing tests remain included because their status is not established. Excluded tests have an evidence-backed reason for exclusion. This claim concerns preserving the full suite’s result while leaving the suite’s ability to detect defects unchanged. It also depends on verdict reliability. Validation must distinguish missed selections from tests that behave inconsistently across runs.
What must happen when reachability cannot be established
Unknown reachability must widen execution. A changed location with no coverage offers no basis for exclusion, so the system must run the tests it would otherwise skip. The same principle applies when the change affects something the execution graph does not describe. Build configuration, CI workflows, dependency manifests, and toolchain definitions can change how tests behave even when they do not show up as ordinary application call paths.
A common policy treats CI and build-tool changes as affecting every test and triggers a full run. The fallback follows the scope of the uncertainty. A fixture, a lockfile or a change outside any function should run the tests it could touch, and when the selector itself fails, every test should run. A failing test must still fail the job.
A Linux-only pull request run cannot settle a reachability question that depends on executing a Windows- or macOS-specific path. Running more tests in an environment that cannot exercise the behavior does not resolve that gap. A team may explicitly accept that some coverage sits outside its pull request gate as a documented trade-off that gives up full-run equivalence.
How the nightly full run fails to be the safety net it is assumed to be
Running selective pull request runs followed by full nightly runs is a valid approach that many teams use. The nightly run provides a periodic audit of the selective system. A regression missed by the pull request gate reaches the main branch before the nightly suite has an opportunity to find it.
The audit can also fail at the reporting boundary. A workflow that dispatches the full suite as another workflow can report success as soon as the dispatch is accepted, and acceptance is not the downstream test result. In that configuration, the apparent safety net has a reporting hole even if the full suite executes.
Map freshness and attention create separate weaknesses. If nightly coverage is not refreshed during quiet weeks, exclusion decisions can continue using a map that no longer reflects the codebase’s structure. Even a correctly reported nightly failure needs a process that brings it to attention comparable to a pull request failure. Neither property follows automatically from scheduling the job. Full nightly runs remain necessary to support a principled pull request gate.
Why agent-written code raises the stakes for reachability-based selection specifically
When agents write both implementation and tests, the tests are written alongside the code and cannot serve as a fully independent check. They can encode the same assumptions and miss the same edge cases as the implementation. Higher pull request velocity increases the pressure to shorten CI at the same time. Ad hoc exclusions compound those pressures by reducing the tests that execute without establishing whether the omitted tests exercise the change.
Reachability analysis establishes that the tests connected to changed behavior are included, even when it cannot judge whether their assertions are useful. That boundary is important for interpreting a green result. Even when the right tests are selected, a suite can have weak tests, and that does not excuse a selector that also skips tests that exercise the change.
Agent changes to CI configuration create a more direct problem because the agent can alter the rules governing its own test run. The workflow becomes part of the change under review. These changes should trigger a full run in place of selective execution and get their own review, however small they seem. This follows the same conservative rule as other infrastructure changes. A code-impact map cannot establish safety for effects outside the relationships it records.
Shadow mode as the discipline for validating a selection system before trusting it
Before a selector gates merges, shadow mode should compare the subset it would run with the full suite. The full run can catch failures the subset would have missed. The key measurement is the escape rate. It turns a claim about reachability into evidence about whether exclusions preserve the verdict under the team’s actual changes.
The comparison needs three outcomes to show more than whether the results agree or disagree. A caught failure occurs when the subset selects a test, it fails, and that failure is visible. An escaped failure occurs when an excluded test fails in the full run and the subset would have passed. An unstable failure occurs when the subset runs a test successfully but that test later fails in the full run.
Reporting the three separately makes diagnosis more useful than a single mismatch count. An escape points to selection and coverage as things to check, while an unstable result points to non-deterministic test behavior.
The three counts only mean something over a stated window: how many test runs the selector left out, how many of those failed in the matching full runs, over how many pipeline runs and which dates, with a statistical bound. Zero escapes on one codebase is not a guarantee for another, which is why each team needs its own shadow period.
An increase in escapes during rapid codebase change, while the map remains unrefreshed, makes staleness visible before exclusions become authoritative. Validation must precede trust in the gate. For suites that are large or change quickly, the shadow period should last until the escape rate reaches an acceptable threshold, even if that takes longer than a calendar deadline allows. That operational threshold does not turn unresolved escapes into proof of equivalence.
What the selection system must show the reviewer for the verdict to be meaningful
A selective verdict needs a visible record of its reasoning. Reviewers must be able to see which tests ran, which were excluded, and the reachability determination behind each exclusion. The record must also show which tests were included to be safe, including new tests, tests that failed before, and tests included because coverage data was missing. If selection triggered a full run, its findings belong in the record too. A green status alone cannot communicate these distinctions.
The need for evidence grows as execution becomes more selective. Stripe reports running about 5% of its Ruby tests per build on average, making the rationale for exclusions a substantial part of the CI artifact. Selection logs stored as CI artifacts, with a summary surfaced on the pull request, give reviewers an inspectable basis for the verdict. The nightly audit needs equally clear reporting. Dispatch acceptance must not stand in for the result of the full suite.
Review evidence can make the affected behavior legible as well. Graphide review reports findings on a pull request and says for each how it is known, including when it could not be verified. Reachability is trustworthy because it has a basis that can be shown, and a system that cannot explain its exclusions cannot be trusted with a gate.


