The retrieval optimization we decided not to ship

Define the gain before comparing candidates.

A retrieval change can improve a local metric and still make the full task worse. It may return a cleaner set of cards while adding work elsewhere, or place a crucial record outside the caller’s preview. We wanted a release decision based on the complete read-and-answer path.

The study completed on 19 October 2026. We compared one candidate-selection change with a frozen baseline against one authorized work-history snapshot. Ten paired questions in the main comparison fed their actual retrieved context into caller continuations. Development, validation, and held-out source families were separated before the comparison.

The candidate needed to improve required-record recall, preserve usefulness and critical information, avoid a complete-task cost increase, and meet a latency gate. These requirements made it possible to reject a candidate that looked promising in selected examples.

Look at the paired results.

The average hid a useful regression.

The candidate improved useful-card coverage on one question. On another held-out question, caller usefulness fell from 3 to 2. The useful record was present in both initial candidate pools, but its final ranking moved from second to thirty-seventh, outside the eight-card preview.

That was not a failure to find the record. It was a failure to deliver it usefully to the caller. Plans and scoring inputs differed between the two runs, so the study did not isolate a pure SQL effect. The finding was still material to the end-to-end decision.

Both variants also omitted critical source constraints on two questions. Those omissions remained even though the required records were found. Record-level recall and complete understanding are different targets, which is why a successful lookup cannot close the quality gate by itself.

Count the complete task.

Estimated cost rose by 10.93% on the measured workload. These were published-rate usage estimates, not verified invoices or cash savings. The accounting included retrieval orchestration, relevance scoring, evidence expansion, and the caller continuation.

One question triggered more scoring work in the candidate: it evaluated another batch even though the required record had already qualified. The configured caps had not increased, but actual work had. A bounded system can still do more unnecessary work within the same limits.

Latency was mixed and uncertain. The read-task median increased while its tail estimate decreased; the small paired sample and broad intervals did not establish the chosen latency gate. We did not select the favorable percentile and call the candidate faster.

What the study can and cannot say.

The result supported holding this candidate on this frozen workload. It did not show that candidate selection is unimportant, that the baseline was complete, or that the current deployed product has the same performance as either measured version.

The comparison used one authorized account, ten one-run pairs, bounded previews, and AI-reviewed usefulness. It did not establish customer outcomes, independent human-expert judgments, unseen tail behavior, or repeat-run stability. Raw history stayed private. An earlier synthetic diagnostic remained separate and did not override the less favorable real-history gates.

The useful lesson is an evaluation boundary: preserve important per-case failures, measure the complete path, and keep the release rule attached to the intended gain. A better subsystem number is evidence to investigate, not permission to ignore a downstream loss.

Reuse the release check for your own context workflow.

Before a retrieval or agent-workflow experiment, write down one must-improve outcome and the behavior that must not regress. Choose questions that require specific constraints, inspect what actually reaches the final answer, and account for all stages the task invokes.

For this study, the decision was to retain the baseline. Keeping that conclusion visible is part of building a context engine whose own engineering history remains useful.

Required gain: what useful outcome should improve?
Preservation: which critical constraints must still reach the caller?
Quality: what would make an answer less useful despite a higher average?
Cost: which complete stages are included?
Latency: what sample and uncertainty can support the decision?
Decision: what result will hold the candidate back?