0.14 successor development screen for output-field projection¶

Tracking issue: #510

Why this exists¶

The output-field projection development screen produced a null result: every factual metric was 0.0000 in every condition, the comparator included. The cause was not the hypothesis. From the shard rows, tool_call_count = 0 and mean turns = 1.21. The frozen B1 agent skipped tool use and fabricated an answer envelope from the tool's own name.

Narrowing further: B1 reached 68–91% task pass on that same runtime and harness with SYSTEM_PROMPT. The projection screen uses FINAL_SYSTEM_PROMPT, which additionally demands a structured answer envelope. So:

Qwen3-0.6B can do tool-calling or a structured answer envelope, not both.

Relaxing the prompt and re-running the same corpus is what #506's Option A governance forbids: prompt or harness semantics is on its frozen list. The sanctioned path is this one: close the screen as consumed and preregister a successor with its own query-disjoint surface.

Instrument eligibility¶

The instrument is chosen by a gate frozen before any candidate runs, because running the roster and keeping the best performer would be choosing the instrument by its outcome, which is the same error one level up.

Criterion	Threshold
episodes emitting a valid answer envelope	≥ 80%
episodes making at least one tool call	≥ 90%
episodes whose answer contains at least one observation-grounded fact	≥ 70%

The third criterion catches a runtime that calls tools and still grounds nothing: it emits envelopes and calls tools, but nothing in the answer traces back to an observation.

Roster, ordered and frozen:

HuggingFaceTB/SmolLM3-3B
Qwen/Qwen3-4B
Qwen/Qwen3-8B

Exact model revisions were frozen before the first qualification episode:

HuggingFaceTB/SmolLM3-3B@a07cc9a04f16550a088caea529712d1d335b0ac1
Qwen/Qwen3-4B@1cfa9a7208912126459214e8b04321603b3df60c
Qwen/Qwen3-8B@b968826d9c46dd6066d109eabc6255188de91218

All qualification candidates use CPU bfloat16 on ubuntu-24.04-arm, Python 3.12.14, PyTorch 2.14.0+cpu and Transformers 4.57.6. SmolLM3 keeps the canonical SDPA + xml_tools path; Qwen candidates use the same OpenAI-function-tool chat-template semantics used by the earlier Qwen development agent. These runtime choices are instrument definition, not experiment outcomes.

The first candidate that qualifies is used; later candidates are not run. Choosing among qualifiers would select the instrument by its outcome, which is the error this gate exists to prevent.

select_runtime in scripts/qualify_agent_utility_runtime.py is the public selection boundary. It does not accept bare rate dictionaries: every candidate must supply provenance-bearing evidence tied to the frozen qualification corpus. Before the ordered-roster rule is applied it verifies the evidence class, surface identity, exact corpus task hash, exact source/harness revision, pinned model revision, exact episode count, and the complete frozen task-ID set. The rate-only selector is private and exists only to unit-test the ordering rule.

Qualification has its own frozen surface, generated by scripts/generate_agent_utility_v8_qualification_corpus.py: one task for every projection-stratum/language pair (36 episodes total), fixed to SR-5 and RAW-FULL. It shares no exact query string with #506 or the successor screen. Its artifact is instrument evidence only; it is never projection evidence.

The candidate/evaluation workflow is implemented in .github/workflows/research-0.14-runtime-qualification.yml. It generates and validates the qualification corpus before candidate execution, records the exact candidate revision and frozen source/harness identity, and passes complete provenance-bearing evidence through select_runtime. A workflow that feeds ad-hoc rows or reimplements the rate comparison remains invalid by construction.

Current qualification execution status — 2026-10-02¶

Infrastructure failures do not advance the frozen roster:

run 36710813766 failed before inference because the qualification corpus artifact was restored under a double-nested path;
run 36808991480 reproduced an evaluator cache-availability failure on later Qwen3-8B shards;
recovery-2 run 36897572955 still had fail-closed evaluator cache restores, so affected Qwen3-8B shards again terminated before inference;
PR #656 removed shared-cache availability as a scientific prerequisite while preserving the exact QUAL_MODEL@QUAL_REV pins, and merge 290f02355570a7db637413d7421fb648f26497b8 automatically launched recovery-3 run 36922946442.

Recovery-3 is an infrastructure-only continuation of the same frozen qualification contract. No partial rows from any invalid attempt are model-quality evidence, and no candidate may be selected until the ordered selector receives complete evidence for the required roster prefix.

Recorded trade-off¶

SmolLM3-3B is also the confirmation arm's runtime, so the screen and the confirmation differ only by surface. This trade-off is accepted: B2's canonical run already shows this model performs the tool-calling task, making it the candidate most likely to qualify first, and a screen that cannot qualify is worth nothing. The screen's surface is query-disjoint, so it does not consume the confirmation surface.

What is identical to #506¶

The four conditions, six distractor strata, six languages, separately reported metrics, paired cluster bootstrap and promotion gate are unchanged, so results stay comparable. Only the runtime and the surface differ.

Query disjointness¶

Enforced at generation, not left to a test. build_corpus raises before writing anything if the surface shares a normalised query with any prior surface. The Issue #506 projection corpus is checked directly, built locally inside the generator rather than registered in scripts/agent_utility_prior_query_guard.py. Registering it there would change known_prior_query_manifest()["union_sha256"], which is stamped into generated v3/v4/v6 corpora and hard-checked by their validators; a stored corpus artifact created before this branch must still validate against main, so that registration was reverted in favour of the local check.

What "disjoint" means here, recorded plainly: this surface is the #506 corpus under a distinct identifier space (S#### in place of P####) — every task is byte-identical to its #506 counterpart apart from that prefix. Disjointness means no shared query string, nothing more; it is not a content-independent surface. That is judged acceptable because #506's screen never called a tool, so no observation content from that corpus ever reached a model, and nothing in this pipeline trains on prior runs. Issue #510's letter ("not one query shared") is met; the surface is not "genuinely different" content.

Governance¶

This screen remains development evidence. It may not be reported as confirmation, and it may not change conditions, task content, distractor strata, thresholds, the scorer, prompt or harness semantics, or row inclusion. #506's corpus, conditions, gate and confirmation arm are untouched.

If no candidate qualifies, that is recorded as a terminal result about the screening approach. It is not a reason to weaken the thresholds.

The workflow executes candidates strictly in roster order. A later candidate may start only after the previous candidate produced complete measured evidence and failed the frozen thresholds. An infrastructure failure (cache failure, runner OOM, model-load failure, missing shard, or workflow failure) is not a scientific non-qualification and stops the chain. The final selector refuses an incomplete failed prefix.

Reproduction¶

Freeze and validate the qualification surface first:

python scripts/generate_agent_utility_v8_qualification_corpus.py \\
  --source-revision "$(git rev-parse HEAD)" \\
  --out artifacts/successor/qualification-corpus.json

python scripts/validate_agent_utility_v8_qualification_corpus.py \\
  --corpus artifacts/successor/qualification-corpus.json

Then generate the successor development surface:

python scripts/generate_agent_utility_v8_successor_corpus.py \
  --source-revision "$(git rev-parse HEAD)" \
  --out artifacts/successor/successor-corpus.json

python scripts/validate_agent_utility_v8_successor_corpus.py \
  --corpus artifacts/successor/successor-corpus.json