Skip to content

SchemaRouter 0.14 paper-evidence checkpoint

Status: 2026-10-10
Scope: evidence that is already terminal or protocol-frozen. Active experiment outcomes are not inferred from partial rows.

This document is a paper-preparation checkpoint for issue #199. It complements docs/research/paper-evidence-package.md and the machine-readable benchmarks/research-experiment-ledger.json. The ledger is the canonical source of experimental facts.

1. Claim hierarchy

The 0.14 cycle separates several evidence levels that must not be collapsed.

Evidence level Work item Permitted interpretation
Retrieval mechanics #418 / #420 Phase A Recall@K and serialized tool-schema compression on the frozen 23-task surface
Small-agent mechanism #420 B1 Controlled sanity/reproducibility evidence for Qwen3-0.6B
Strong-agent replication #423 B2 Evidence that the mechanism is not specific to the small Qwen3 runtime
Optimization ablations #430 and structural K3/K5 Candidate-depth / representation optimization only; failed candidates are not repaired post hoc
Corrective retrieval #431 Whether typed observable execution state improves bounded re-retrieval; preregistered gate resolved, condition not promoted
Large held-out generalization #432 Required before a broad heterogeneous-task/language/catalog generalization claim
Final-answer factual quality #424 Required before claiming preserved/improved factual answer quality, units, or provenance
Output-field projection #506 / #510 Separate field-level question; current successor runtime qualification is instrument evidence, not projection evidence

The standing rule is: a DEV or mechanism pass is not a generalization claim, and an infrastructure failure is not a scientific negative result.

2. Research-question and hypothesis table

Work item Research question / hypothesis Evidence role Current state
#420 B1 Does bounded Top-K capability exposure preserve task completion while sharply reducing tool-schema context relative to FULL? controlled mechanism / sanity terminal
#423 B2 Does the same fixed experiment reproduce on a materially stronger non-Qwen agent runtime? strong-agent replication terminal success
#434 Can typed multi-field / intent-aware representations improve retrieval without unacceptable latency? development ablation terminal, no promotion
#430 adaptive depth Can adaptive shortlist depth beat fixed K while preserving required-tool coverage? development optimization terminal negative
structural fixed K3 Can structural Top-3 preserve downstream task success while reducing schema context relative to structural Top-5? strong-agent optimization terminal negative for promotion
#431 Can re-retrieval from typed observable execution state improve multi-step capability recovery without changing execution authority? successor experiment terminal gate resolved; no post-result retuning
#432 Does the promoted fixed protocol generalize on 780 independently authored held-out tasks? held-out confirmation terminal canonical negative; 38501112190 verified 11,700 episodes; all SR non-inferiority gates failed
#424 Does capability-context reduction preserve final-answer facts, values, units and provenance? final-answer confirmation canonical 38506709595 completed; valid negative: all deployable primary gates failed
#506 Does output-field projection preserve answer factual quality while reducing observation context? field-level experiment original DEV invalid as answer instrument; separate frozen 38564885951 confirmation terminal, projection promotion negative (unsupported-fact gate)
#510 Can a preregistered stronger runtime qualify to measure the #506 successor screen without outcome-based model shopping? instrument qualification terminal negative; selected runtime null, invalid attempts excluded

3. Dataset / surface roles

Surface Role Tuning allowed? Status / rule
#420 B1 frozen 23-task surface controlled mechanism evidence no post-result semantic retuning consumed, terminal
#423 B2 same semantic surface strong-agent replication no consumed, terminal
#434 DEV representation development only within preregistered DEV process consumed; no candidate promoted
#434 confirmation confirmation no sealed / unconsumed because DEV did not promote
#430 adaptive DEV development optimization only by preregistered family/rule consumed; terminal negative
structural K3/K5 downstream surface promotion gate no consumed; K3 not promoted
#431 180-task surface successor experiment no row-level post-hoc retuning canonical gate terminal; not promoted
#432 780-task surface large held-out confirmation no frozen canonical aggregate complete, negative product/non-inferiority gates
#424 144-task surface final-answer confirmation no independently scored; valid terminal negative result
#506 original projection DEV development screen no repair from outcome consumed; no tool calls, instrument too weak
#510 36-episode qualification instrument qualification only no experiment tuning terminal negative; invalid cache/artifact attempts excluded
#510 successor projection surface successor DEV evidence no post-hoc condition/scorer changes no qualified runtime; successor unstarted

4. Frozen candidate/runtime configurations

B1 canonical runtime

  • model: Qwen/Qwen3-0.6B
  • exact runtime smoke: Python 3.12.14, PyTorch 2.14.0+cpu, Transformers 4.57.6, tokenizers 0.22.2, safetensors 0.8.0
  • semantic tasks: 23
  • catalogs: 20 / 50 / 100 / 250
  • accepted aggregate: 552 unique (catalog_size, task_id, condition) episodes
  • canonical task SHA-256: bc0b78ff2be11b89e6ac54ea0ee336f944f04b3c203fc61da70a46ff48b4e03c

B2 canonical runtime

  • model: HuggingFaceTB/SmolLM3-3B
  • model revision: a07cc9a04f16550a088caea529712d1d335b0ac1
  • runner: ubuntu-24.04-arm
  • attention: PyTorch SDPA
  • semantic tasks: 23
  • catalogs: 20 / 50 / 100 / 250
  • conditions: FULL / SR-5 / SR-10 / SR-PROGRESSIVE / ORACLE
  • aggregate: 460 episodes
  • canonical run: 36642658406
  • partial outcomes from attempts 1-8 are excluded from tuning and final inference

#510 runtime qualification

Frozen roster, in order:

  1. HuggingFaceTB/SmolLM3-3B@a07cc9a04f16550a088caea529712d1d335b0ac1
  2. Qwen/Qwen3-4B@1cfa9a7208912126459214e8b04321603b3df60c
  3. Qwen/Qwen3-8B@b968826d9c46dd6066d109eabc6255188de91218

Frozen qualification runtime:

  • ARM64 CPU, bfloat16
  • Python 3.12.14
  • PyTorch 2.14.0+cpu
  • Transformers 4.57.6
  • tokenizers 0.22.2
  • safetensors 0.8.0
  • seed 20260929
  • max new tokens 256
  • max turns 6
  • threads 4
  • 36 episodes = 6 projection strata × 6 languages
  • catalog 100
  • SR-5 candidate exposure
  • RAW-FULL observation condition
  • selection rule: first qualifying candidate in the frozen roster order
  • eligibility: envelope >= 0.80, tool call >= 0.90, grounded fact >= 0.70

An episode receives grounded-fact credit only when it both calls a tool and returns at least one required fact grounded in the observation.

5. Terminal metrics currently suitable for paper tables

B1 controlled small-agent result

Canonical workflow: 36529108855

Condition / comparison Result
FULL task pass 68.48%
SR-3 task pass 82.61%
SR-5 task pass 91.30%
SR-10 task pass 81.52%
SR-PROGRESSIVE task pass 82.61%
ORACLE task pass 86.96%
SR-5 minus FULL +22.83 pp
clustered bootstrap interval +9.78 pp to +36.96 pp
SR-5 required-route Recall 100%
SR-5 all-required task coverage 100%
SR-5 mean tool-schema tokens 1,315.9
FULL mean tool-schema tokens 24,269.6
SR-5 / FULL schema-token ratio 5.42%
unauthorized destructive executions 0

Claim boundary: this is a 23-semantic-task controlled Qwen3-0.6B mechanism result, not population-level non-inferiority, all-agent generalization, or final-answer-quality evidence.

Structural K3 vs K5 downstream result

Canonical workflow: 36670280971

Metric STRUCT-FIXED-3 STRUCT-FIXED-5
task pass 82.6087% 85.8696%
mean tool-schema tokens 1,040.89 1,367.17
unauthorized destructive executions 0 0

Paired K3-K5 task-pass delta: -3.2609 pp.
Preregistered non-inferiority floor: -2 pp.
Task-cluster bootstrap 95% CI: [-13.0435 pp, +3.2609 pp].

The schema-token and execution-policy gates passed, but the paired task-pass gate failed. K3 is a scientific negative for promotion and is not included in the #432 held-out condition manifest.

6. Provenance table

Evidence Source / run Artifact / digest Status
B1 canonical source b9eadefd3cd076f026a54bbc55a949f0424f5dab, run 36529108855 artifact 11021506964; digest sha256:2e101d62dbe7f3202a24f0f40c49000064991b1521c4154c8feeb587de510271; aggregate SHA 33678700298a47b451ea1f03377cd874a1a1e373597cf961d207e07ae39f568d terminal
B2 canonical source 01edb00fe7e8bff803988ce6bce5e05f79801e43, run 36642658406 canonical digest sha256:edbccbbfb44d58ba936af7af82efe844177edc24dd587e3c52cff1505c37c256 terminal success
structural K3/K5 run 36670280971 artifact 11081310428; digest sha256:ae7fb24ac365d307a52b07e416dff6308581ccca96a4831d8c41f44f5170b613 terminal negative for K3 promotion
#431 corrective frozen source 30663de8f618bc88a893d9bf6214035a70e8e894; parent 36931249518 + recovery 37793749175 canonical artifact checked by #500 conveyor; optional condition false terminal gate; not promoted
#510 qualification attempt 1 merge/source 5c0b827a6ca699bb2d6de8982820829ff67cacd4; run 36710813766 no candidate evidence produced infrastructure-invalid: artifact restore path failed before inference
#510 cache recovery attempts runs 36808991480 and 36897572955 incomplete Qwen3-8B evidence only; do not consume infrastructure-invalid: evaluator cache availability terminated shards before inference
#510 recovery-3 merge 290f02355570a7db637413d7421fb648f26497b8; run 36922946442 selection artifact 11206309688; SHA-256 55ed4c42b459c1807ab7005a930c08b608aab1fc43eae1a76662b86e4f85d32c terminal negative; no qualified runtime

7. Invalidated / infrastructure-only run appendix

These runs must remain visible, but must not be cited as model-quality failures.

Run Classification Reason
B1 36517371433, 36520025834, 36520831273 superseded / invalidated pre-v2 task/protocol lineage; excluded from accepted B1 inference
B2 36641753066 noncanonical duplicate launch concurrency cancelled the earlier semantically identical attempt-9 launch; partial rows not reused
K3 36668831941 infrastructure-invalid repository-local benchmarks import failure before inference
K3 36669634556 infrastructure-invalid pinned SmolLM3 cache miss before accepted scientific evidence
#431 36669636302 infrastructure-invalid repository-local scripts import failure; stale-wrapper retry also invalid
#431 36670710985 infrastructure-invalid terminal run later evaluator shards lost the shared model cache; no terminal aggregate may be interpreted
#510 36710813766 infrastructure-invalid qualification corpus artifact restored under a double-nested path; all SmolLM3 shards failed before inference, so no candidate rates/aggregate/selection exist
#510 36808991480 infrastructure-invalid Qwen3-8B evaluator shards failed on cache restore before inference; rerun reproduced the same transport failure
#510 36897572955 infrastructure-invalid explicit warm-up cache publication was insufficient while evaluators still failed closed on cache miss; affected Qwen3-8B shards terminated before inference
#506 DEV 36682589574 valid development instrument failure, not projection comparison tool_call_count=0 and comparator factual metrics were also zero; the runtime did not measure the projection question
#506 original confirmation 38564885951 terminal valid negative promotion decision; original frozen 3B arm, not #510 successor 96/96 shards and 1,152 episodes; artifact 11922570109, ZIP SHA-256 c11912ed67a3c02e90c5963ccb462ab44fba3dbd28ced0fe9aacaa92d390eff7; PROJECTED unsupported-fact delta +14.58pp (paired 95% CI [+8.68,+20.83]pp) against frozen +1pp gate; RAW-FULL absolute fact recall 0%, so broad causal claim unqualified

The #506 entry is intentionally distinct from infrastructure failures: the workflow ran, but the small agent failed the prerequisite measurement behavior. Its result is evidence about instrument suitability, not evidence for or against projection quality.

8. Freeze-state matrix

Work item Protocol frozen? Content frozen? Inference authorized? Outcome available?
B1 yes yes completed yes
B2 yes yes completed yes
#430 adaptive yes yes completed yes
structural K3/K5 yes yes completed yes
#431 yes yes completed terminal gate: not promoted
#432 yes yes completed, canonical run 38501112190 valid terminal negative, all SR arms fail gate
#424 yes authoring/validation contract frozen completed run 38506709595 canonical negative; no deployable arm passed
#506 original DEV yes yes completed instrument failure; distinct original strong confirmation 38564885951 terminal negative promotion
#510 qualification yes yes completed terminal negative; runtime null
#510 successor projection yes frozen successor semantics blocked; qualification failed no

9. Authority / verifier architecture

flowchart TD
    A["User request"] --> B["Registered OpenAPI / MCP / Python contracts"]
    B --> C["Typed capability registry"]
    C --> D["Bounded capability retrieval"]
    D --> E["Schema, arguments, authorization and policy checks"]
    E --> F["Authorized tool execution"]
    F --> G["Typed observations / field projection"]
    G --> H["Final answer"]
    G -. "Corrective retrieval from observed evidence" .-> C

The model never creates an executable route, field, argument authority, or pseudo-capability. Corrective retrieval may change candidate exposure from observable typed execution state, but execution authority remains local and registered.

10. Reproducibility manifest

Before using a row/result in a paper table, recover and verify:

  1. work item and experimental role;
  2. source revision;
  3. preregistration / freeze manifest;
  4. corpus/task identity and SHA-256 where applicable;
  5. exact model and immutable model revision;
  6. Python / framework / inference runtime identity where materially relevant;
  7. workflow run ID;
  8. artifact ID and digest;
  9. aggregate identity and expected episode/task count;
  10. terminal decision and claim boundary;
  11. invalidated/superseded predecessors;
  12. whether calibration, confirmation, or blind evidence was consumed.

Canonical repository sources:

  • benchmarks/research-experiment-ledger.json
  • docs/research/design-and-experiment-history.md
  • docs/research/paper-evidence-package.md
  • frozen manifests under benchmarks/
  • terminal GitHub issue / PR comments with run, artifact and digest provenance

11. Pending evidence before paper closure

The following must remain explicitly incomplete:

  • Issue #431 terminal aggregate and its preregistered promotion boolean;
  • the automatically frozen #432 held-out condition manifest;
  • complete #432 780-task held-out aggregate and statistical intervals;
  • Issue #424 final-answer factual/value/unit/provenance aggregate: terminal valid negative, artifact 11888156120, SHA-256 c0593143deb4a06a1433c8bbfdbeab28a23e45156ea6e0c2c5744386cddb83cb; coverage SR-5/SR-10/SR-PROGRESSIVE 51.389%/50.000%/46.181%; unsupported facts worsen vs FULL; no deployable condition passes.
  • Issue #510 runtime-qualification terminal selector after infrastructure-only recovery of attempt 36710813766;
  • Issue #510 successor projection result and later confirmation, if its gates authorize one;
  • complete per-language / per-route / per-family paper slices for the final promoted line;
  • any final calibration/blind evidence whose protocol is still blocked.

No pending item should be filled from partial workflow rows or inferred from a running job state.