Complete experiment index¶
This page exposes the full experiment ledger for SchemaRouter's routing research rather than only the latest or strongest results.
The public summary page answers "where the research currently stands." This page answers a different question: what was actually tried?
Current machine-readable ledger:
- independent experiment records: 92;
- legacy routing corpus lineage: 13 versioned corpora;
- routine bugfix-only commits are not counted as independent experiments unless they changed an architecture invariant, evaluation protocol, or empirical claim;
- failed, superseded, invalidated, and terminal experiments are retained rather than hidden.
The 92-record count includes the terminal 0.13 V6A–V6H/open-set controls and the active 0.14 agent-utility lineage. Terminal 0.13 confirmation surfaces remain unopened unless explicitly recorded otherwise.
The canonical machine-readable source is
benchmarks/research-experiment-ledger.json.
The narrative source is
Design and experiment history.
For what to try next and why, use the Prior-art roadmap and active GitHub issue #417. Historical 0.13 prior-art mapping remains in #388. Those surfaces map literature to canonical work items and record active/next/backlog/deferred state so a new session does not recreate terminal experiments.
Choose the right evidence surface¶
Use this ledger when you need provenance for a specific experimental decision. For ordinary reading, start with the smaller surfaces below:
| Question | Start here |
|---|---|
| What is the current defensible research claim? | Research status |
| What evidence is paper-ready? | Paper evidence package |
| What is still active or blocked? | Prior-art roadmap and the active research issues |
| What exactly was tried, including failures? | This complete ledger |
| How did the architecture evolve? | Design and experiment history |
The sections below are intentionally archival. Historical failures are retained for reproducibility; they are not recommended product configurations.
How to read the evidence¶
A Git commit is not the same thing as an experiment. One squashed PR may contain implementation, tests, preregistration, corpus freeze, calibration, and final evidence; conversely, one experiment may require several commits or workflow runs. The ledger therefore treats the experimental decision as the unit of record and links it back to issues, PRs, revisions, workflows, artifacts, and source files when available.
The repository Git history remains the exhaustive engineering record.
Legacy corpus lineage¶
| Corpus | Role |
|---|---|
decision-routing-v1.json |
Earlier routing / holdout lineage retained in Git history and later research governance |
decision-routing-v2.json |
Earlier routing / holdout lineage retained in Git history and later research governance |
decision-routing-v3.json |
Earlier routing / holdout lineage retained in Git history and later research governance |
decision-routing-v4-operation-holdout.json |
Earlier routing / holdout lineage retained in Git history and later research governance |
decision-routing-v5-operation-calibration.json |
Earlier routing / holdout lineage retained in Git history and later research governance |
decision-routing-v6-operation-holdout.json |
Earlier routing / holdout lineage retained in Git history and later research governance |
decision-routing-v7-operation-post-change-holdout.json |
Earlier routing / holdout lineage retained in Git history and later research governance |
decision-routing-v8-operation-alias-holdout.json |
Earlier routing / holdout lineage retained in Git history and later research governance |
decision-routing-v9-operation-alias-holdout.json |
Earlier routing / holdout lineage retained in Git history and later research governance |
decision-routing-v10-operation-generalization-holdout.json |
Earlier routing / holdout lineage retained in Git history and later research governance |
decision-routing-v11-operation-generalization-holdout.json |
Earlier routing / holdout lineage retained in Git history and later research governance |
decision-routing-v12-operation-contrastive-holdout.json |
Earlier routing / holdout lineage retained in Git history and later research governance |
decision-routing-v13-operation-contrastive-holdout.json |
Earlier routing / holdout lineage retained in Git history and later research governance |
Legacy / pre-cycle¶
| # | Experiment | Decision | Hypothesis / purpose | Evidence |
|---|---|---|---|---|
| 1 | pre-0.8-no-route-sentinel-ablation |
sentinel_removed | An explicit none_of_the_above pseudo-option could improve empty-recall no-route discrimination. | issue #87 · e41f57a044 |
0.10-operation-contrastive-v1¶
| # | Experiment | Decision | Hypothesis / purpose | Evidence |
|---|---|---|---|---|
| 1 | 0.10-contrastive-bge-v1 |
blind_final_passed_for_optional_profile | Sibling-contrastive BGE operation-fit can improve supported operation routing while retaining near-domain unsupported rejection. | ledger / Git history |
0.10-operation-cascade-v2¶
| # | Experiment | Decision | Hypothesis / purpose | Evidence |
|---|---|---|---|---|
| 1 | 0.10-cheap-first-cascade-v2 |
rejected_on_calibration | A MiniLM fast-path plus selective BGE reranking can reduce CPU latency while preserving preregistered quality floors. | ledger / Git history |
0.10-operation-graph-projection-v3¶
| # | Experiment | Decision | Hypothesis / purpose | Evidence |
|---|---|---|---|---|
| 1 | 0.10-graph-projection-v3 |
rejected_on_fresh_calibration | A typed schema graph can own routing authority while semantic signals provide bounded soft evidence, reducing latency without sacrificing quality/safety. | PR #181 |
0.11-operation-routing-quality-v4¶
| # | Experiment | Decision | Hypothesis / purpose | Evidence |
|---|---|---|---|---|
| 1 | 0.11-v4-fresh-baseline |
baseline_only | Fresh natural-language development data can expose the post-0.10 routing bottleneck without reusing consumed calibration/blind evidence. | PR #187 |
| 2 | 0.11-hierarchical-tool-operation |
rejected_on_development | Separating tool-domain selection from within-tool operation selection will improve exact-route accuracy while preserving rejection. | PR #189 |
| 3 | 0.11-accepted-operation-selector |
standalone_candidate_rejected_primitive_retained | If operation-fit already identifies a registered route, allowing that accepted route to control the single-call candidate will recover endpoint accuracy without changing the abste… | PR #190 |
| 4 | 0.11-stage-signal-diagnostics |
diagnostic_supports_multi_signal_boundary_design | Candidate-fit score geometry can reveal which supported misses are recoverable and whether a global similarity threshold can separate supported from near-domain unsupported reques… | 1cf7efc261 |
| 5 | 0.11-typed-evidence-infrastructure |
infrastructure_retained_for_next_candidate | Recorded design / evaluation step. | ledger / Git history |
| 6 | 0.11-route-local-threshold-ablation |
rejected_on_development | Per-route pairwise minimum-score thresholds can recover open-set safety while preserving the operation-narrow supported-routing gain. | PR #191 |
| 7 | 0.11-nli-boundary-superseded |
superseded_before_result_interpretation | Recorded design / evaluation step. | PR #192 |
| 8 | 0.11-pairwise-tool-hierarchy |
rejected_on_development | Pairwise tool ranking crossed the 70% supported gate but did not provide an open-set capability boundary and badly missed rejection/false-route gates. | PR #193 |
| 9 | 0.11-direct-multilingual-nli-boundary |
rejected_on_development | Conservative entailment thresholds improved unsupported rejection but collapsed supported recall; direct NLI is unsuitable as the operation selector/gate tested here. | PR #194 |
| 10 | 0.11-bounded-retrieve-rerank-architecture |
architecture_implemented_diagnostic_pending | Recorded design / evaluation step. | PR #195 · 90af3e1fce |
| 11 | 0.11-bounded-rerank-score-diagnostic |
ranking_headroom_validated_latency_optimization_required | Recorded design / evaluation step. | PR #205 · f43535b6ef |
| 12 | 0.11-winner-only-threshold-semantics |
mechanism_merged_into_active_research_stack | Recorded design / evaluation step. | PR #208 |
| 13 | 0.11-evidence-projector-active-stack |
infrastructure_merged_into_active_research_stack | Recorded design / evaluation step. | ledger / Git history |
| 14 | 0.11-bounded-rerank-width-ablation |
selected_width_2_by_preregistered_smallest_passing_width_rule | Recorded design / evaluation step. | PR #216 · 9f4dbff423 |
| 15 | 0.11-joint-score-margin-boundary |
not_selected | Joint score+margin gating adds only ~0.35 percentage points at the same false-route budget, insufficient to justify a larger route-local tuning surface before width/latency optimi… | ledger / Git history |
| 16 | 0.11-width2-winner-gate-executable |
rejected_on_development_latency_gate | A width-2 bounded semantic recall plus score-only BGE rank-then-gate boundary can satisfy the 0.11 development quality gates without material latency regression. | PR #228 · fdaf3f77e9 |
| 17 | 0.11-cheap-action-only-evidence-diagnostic |
retain_as_cheap_bounded_selector_or_auxiliary_evidence_not_global_direct_fast_path | A cheap multilingual embedding over endpoint action names and trusted operation aliases can provide independent routing evidence at materially lower latency than BGE. | PR #230 · 521dcc65e0 |
| 18 | 0.11-action-guided-single-pair-bge-diagnostic |
rejected_on_supported_recall | Use cheap action-only evidence to choose one already-authorized width-2 candidate, then score only one BGE pair to remove CPU tail latency. | issue #231 · PR #232 |
| 19 | 0.11-bounded-action-embedding-diagnostic |
rejected_on_open_set_supported_recall | Replace BGE inside width-2 authorized recall with cached action-only MiniLM embedding evidence. | issue #233 · PR #236 |
| 20 | 0.11-minilm-dual-view-diagnostic |
rejected_backbone_capacity_insufficient | Use one MiniLM query embedding against cached schema/domain and action-only route views to separate ranking evidence from open-set confidence. | issue #240 · PR #241 |
| 21 | 0.11-multilingual-embedding-backbone-screen |
bge_m3_selected_for_strict_open_set_optimization | Keep the dual-view routing architecture fixed and change only multilingual embedding backbone capacity. | issue #242 · PR #243 |
| 22 | 0.11-gte-winner-bge-rejector |
rejected_on_supported_recall | Let high-capacity GTE choose one winner and use one BGE cross-encoder pair only as the unsupported-operation rejector. | issue #244 · PR #249 |
| 23 | 0.11-bge-m3-budget6-executable-confirmation |
not_promoted_literal_freeze_missed_by_one_case | One supported papers.citations case scored ~1.35e-7 below its frozen threshold; all planner/direct parity and safety gates passed, but the literal 960-correct frozen projection re… | issue #245 · PR #247 |
| 24 | 0.11-bge-m3-fine-fusion-strict |
no_fusion_weight_reached_85_percent_strict_target | Best preregistered strict point was 83.77% exact at 98.96% near-domain rejection and 0.93% false-route. No interpolation or out-of-grid retuning was permitted. | issue #246 · PR #248 · 9d6f5bdc18 |
| 25 | 0.11-conditional-zero-false-rejected-winner-rescue |
rejected_zero_false_rescue_recovers_only_5_of_required_15 | Recorded design / evaluation step. | issue #255 · PR #258 |
| 26 | 0.11-route-local-stable-bge-m3-fusion |
rejected_strict_boundary_below_85_percent_despite_90_54_raw_ceiling | Route-local fusion weights selected from the already-preregistered #246 grid may recover the remaining exact-route gap while midpoint/canonicalized thresholds remove observed-scor… | issue #256 · PR #257 |
| 27 | 0.11-numerically-robust-threshold-freeze-policy |
required_for_future_frozen_candidates | Recorded design / evaluation step. | issue #253 |
| 28 | 0.11-bge-m3-robust-budget6-executable-confirmation |
confirmed_as_numerically_robust_strict_base_for_conditional_rescue | Recorded design / evaluation step. | issue #259 · 9e9b1049ea |
| 29 | 0.11-cross-model-zero-false-abstention-rescue |
gte_agreement_bge_selected_for_frozen_confirmation | Recorded design / evaluation step. | issue #262 · PR #263 |
| 30 | 0.11-zero-false-cross-model-frozen-candidate |
rejected_surface_fragile_positive_open_set_boundary | Recorded design / evaluation step. | issue #265 · PR #270 |
| 31 | 0.11-robust-base-winner-crossencoder-rescue |
rejected_safe_but_insufficient_rescue | Recorded design / evaluation step. | issue #266 |
| 32 | 0.11-native-bge-m3-abstention-geometry-rescue |
rejected_native_geometry_insufficient | Recorded design / evaluation step. | issue #271 |
| 33 | 0.11-typed-contradiction-only-nli-veto |
rejected_generic_nli_neutral_on_supported_and_unsupported | Recorded design / evaluation step. | issue #273 |
| 34 | 0.11-explicit-negative-capability-prototype-veto |
rejected_combined_winner_domain_ood_veto | Recorded design / evaluation step. | issue #275 · PR #276 |
| 35 | 0.11-global-signed-capability-prototype-bank |
rejected_global_signed_prototype_evidence_too_coarse | Recorded design / evaluation step. | issue #277 · PR #278 |
| 36 | 0.11-dual-signed-negative-openworld |
rejected_scalar_signed_evidence_overlap | Recorded design / evaluation step. | issue #279 · PR #280 |
| 37 | 0.11-rank-based-capability-set-openworld |
rejected_fixed_prototype_heuristics_terminated | Recorded design / evaluation step. | issue #281 · PR #283 · 944df2e0f2 |
| 38 | 0.11-grouped-oof-learned-winner-verifier |
promote_hgb_p0_500_to_separate_frozen_candidate | Recorded design / evaluation step. | issue #285 · PR #286 · cfaafb84bb |
| 39 | 0.11-external-qwen3-semantic-capability-verifier |
reject | Recorded design / evaluation step. | issue #289 · PR #290 · 68e812ab5c |
| 40 | 0.11-frozen-hgb-winner-verifier-candidate |
rejected_fresh_surface_generalization_failure | Recorded design / evaluation step. | issue #287 · PR #288 · e5b10ee01e |
| 41 | 0.11-system-one-provider-contract |
accepted_reusable_infrastructure | Recorded design / evaluation step. | issue #291 · PR #292 · 7f4118ea05 |
| 42 | 0.11-pluggable-system-one-routing |
direct_laya_rejected_parent_remains_open_for_system_one_comparison | Recorded design / evaluation step. | issue #293 · PR #294 · 48e329ee94 |
| 43 | 0.11-kev-08b-choice-noul-screen |
no_quality_result_runtime_nonviable | Recorded design / evaluation step. | issue #299 · PR #300 |
| 44 | 0.11-pinned-laya-noul-veto |
reject | Recorded design / evaluation step. | issue #301 · PR #302 · 46af3c3d01 |
| 45 | 0.11-laya-choice-noul-full-catalog |
superseded_by_301 | Recorded design / evaluation step. | issue #295 · PR #296 |
| 46 | 0.11-top4-typed-capability-routing |
same_laya_variant_dominated; provider_neutral_contingency_only | Recorded design / evaluation step. | issue #303 · PR #310 |
| 47 | 0.11-generic-decision-callable-benchmark |
accepted_reusable_infrastructure | Recorded design / evaluation step. | issue #304 · PR #305 · c9678b95a6 |
| 48 | 0.11-anyjev-l0-content-free-noul-veto |
await_realistic_runtime_before_execution | Recorded design / evaluation step. | issue #311 · PR #313 |
| 49 | 0.11-bge-plus-kev-global-noul |
source_kev_analysis_missing | Recorded design / evaluation step. | issue #314 |
| 50 | 0.11-lightweight-negative-gte-offline-compose |
pass_promote_executable | Recorded design / evaluation step. | issue #322 · PR #323 |
| 51 | 0.11-lightweight-negative-gte-executable |
pass_dev_promote_fresh | Recorded design / evaluation step. | issue #324 · PR #325 |
| 52 | 0.11-lightweight-bge-gte-frozen-fresh-confirmation |
reject_after_fresh_failure | Recorded design / evaluation step. | issue #326 · PR #327 · b19d7b0255 |
| 53 | 0.11-bge-m3-colbert-operation-contract |
reject | Recorded design / evaluation step. | issue #328 · PR #329 · 4c72f2dd19 |
| 54 | 0.11-bge-registry-alias-envelope |
reject | Registry-derived sibling contrast preserves recall but accepts unsupported requests; registry cohesion rejects unsupported requests only by collapsing supported exact to ~24% and … | issue #332 · PR #333 · fa091f4329 |
| 55 | 0.11-bge-gte-threshold-free-consensus |
reject | Cross-backbone top-1 agreement behaves as route-selection confidence rather than unsupported-capability evidence: two strong rankers frequently agree on the same plausible registe… | issue #336 · PR #337 · d25f427569 |
| 56 | 0.11-registry-compiled-capability-verifier |
terminal_rejected_overconservative_veto | A provider-neutral capability IR plus generic synthetic counterfactual supervision can veto unsupported operations for arbitrary newly registered native/OpenAPI/MCP tools without … | issue #338 · PR #341 · ef75100abc |
0.12-query-first-typed-capability¶
| # | Experiment | Decision | Hypothesis / purpose | Evidence |
|---|---|---|---|---|
| 1 | 0.12-query-first-typed-frame-v1 |
terminal_rejected_insufficient_open_set_rejection | A registry-independent explicit request frame plus deterministic within-tool contract filtering can turn same-domain unsupported operations into empty capability sets without lear… | issue #347 · PR #348 · ef0e0a567f |
0.12-semantic-action-ontology¶
| # | Experiment | Decision | Hypothesis / purpose | Evidence |
|---|---|---|---|---|
| 1 | 0.12-semantic-action-ontology-v1 |
terminal_rejected_flat_semantic_argmax | A frozen registry-independent multilingual semantic action ontology can classify request operation semantics before deterministic tool-local capability matching without learned th… | issue #349 · PR #352 · 38ee755756 |
0.12-hierarchical-capability-ontology¶
| # | Experiment | Decision | Hypothesis / purpose | Evidence |
|---|---|---|---|---|
| 1 | 0.12-hierarchical-capability-ontology-v1 |
terminal_rejected_hard_ontology_filter | A hierarchical executable-capability ontology with root/leaf constraints and contrastive multilingual prototypes can improve open-set membership while preserving registered execut… | issue #354 · PR #357 · 250845bba0 |
0.12-asymmetric-ontology-veto¶
| # | Experiment | Decision | Hypothesis / purpose | Evidence |
|---|---|---|---|---|
| 1 | 0.12-asymmetric-ontology-veto-v1 |
terminal_rejected_high_precision_low_recall_asymmetric_veto | Preserve the frozen BGE-M3 raw top-1 as the sole positive route selector and use ontology evidence only as an asymmetric unsupported-membership veto under a fixed independent-agre… | issue #358 · PR #360 · 759359882c |
0.12-capability-set-membership-veto¶
| # | Experiment | Decision | Hypothesis / purpose | Evidence |
|---|---|---|---|---|
| 1 | 0.12-capability-set-membership-consensus-v1 |
terminal_rejected_membership_consensus_insufficient | Replace exact unsupported-leaf agreement with anchored-tool capability-set membership consensus while preserving frozen BGE-M3 as the sole positive route selector. | issue #363 · PR #364 · f198f896c4 |
0.12-external-zeroshot-membership¶
| # | Experiment | Decision | Hypothesis / purpose | Evidence |
|---|---|---|---|---|
| 1 | 0.12-external-multilingual-zeroshot-membership-v1 |
terminal_rejected_multiclass_outside_label_no_open_set_boundary | A small externally pretrained multilingual zero-shot classifier can judge whether a request belongs to the finite capability set registered for the BGE-anchored tool, while remain… | issue #371 · PR #372 · 5e6dde0c38 |
0.12-set-conditioned-binary-entailment¶
| # | Experiment | Decision | Hypothesis / purpose | Evidence |
|---|---|---|---|---|
| 1 | 0.12-set-conditioned-binary-entailment-v1 |
terminal_rejected_all_not_entailment | Use one direct NLI sequence-pair judgment over the anchored tool's full registered capability set; preserve the frozen BGE-M3 winner on entailment and veto to NO_ROUTE on not-enta… | issue #374 · PR #375 · e61058d0aa |
0.12-independent-capability-entailment¶
| # | Experiment | Decision | Hypothesis / purpose | Evidence |
|---|---|---|---|---|
| 1 | 0.12-independent-per-capability-entailment-v1 |
terminal_rejected_independent_entailment_over_veto | Judge every registered capability leaf independently with a pinned multilingual NLI model; preserve frozen BGE-M3 raw top-1 if any registered leaf is entailed, otherwise veto to N… | issue #377 · PR #379 · a872c9602d |
0.12-pairwise-nli-membership¶
| # | Experiment | Decision | Hypothesis / purpose | Evidence |
|---|---|---|---|---|
| 1 | 0.12-pairwise-supported-counterfactual-nli-v1 |
terminal_rejected_pairwise_nli_membership_and_latency | Compare maximum independent NLI entailment over the anchored tool's registered capability leaves against maximum entailment over counterfactual tool/non-tool leaves; veto only whe… | issue #378 · PR #380 · 02aeefd465 |
Why the summary page shows fewer rows¶
Routing research status is a current-state summary. It shows the standing target, strongest reference points, decisive fresh-confirmation failures, and the present interpretation. It is not meant to replace the complete ledger.
For full reconstruction, use all three surfaces:
- this experiment index for the complete catalog;
- Design and experiment history for architectural chronology;
- the machine-readable ledger and Git history for exact provenance.
This separation keeps the main documentation readable without erasing negative results or abandoned branches.
0.13 schema-derived open-set membership¶
| # | Experiment | Decision / state | Purpose / interpretation | Evidence |
|---|---|---|---|---|
| 1 | 0.13-v6a-schema-adb |
terminal_dev_quality_fail | Schema-only positive views can define endpoint-local adaptive spherical regions that preserve supported natural-language requests while rejecting unsupported requests. | issue #384 |
| 2 | 0.13-v6b-hard-negative-ellipsoid |
terminal_dev_quality_fail | Same-resource hard negatives from the registered capability complement plus a low-rank anisotropic ellipsoid can preserve natural supported requests while rejecting near-domain OOS. | issue #395 |
| 3 | 0.13-v6c-tied-gaussian-density-ratio |
terminal_dev_quality_fail | A tied diagonal Gaussian likelihood ratio between schema positives and complement negatives can avoid absolute-boundary collapse while remaining veto-only. | issue #397 |
| 4 | 0.13-v6d-component-gaussian-mixture |
terminal_dev_quality_fail_pr_closed_unmerged | Registry-fixed endpoint/complement Gaussian components with log-sum-exp mixture evidence can model capability multimodality without changing positive route authority. | issue #399 |
| 5 | 0.13-v6e-knn-membership |
terminal_dev_quality_fail_pr_closed_unmerged | A threshold-free non-parametric k=3 local-neighborhood comparison over schema-positive, complement, and frozen generic background banks can avoid Gaussian assumptions and improve open-set membership. | issue #401 |
| 6 | 0.13-naturalistic-operation-probe-membership |
terminal_dev_quality_and_runtime_fail_pr_closed_unmerged | Naturalistic multilingual linear probes improve broad OOD recognition but remain insufficient for same-domain unsupported-operation membership and supported-route preservation. | issue #404 · PR #405 · workflow 36499927289 |
| 7 | 0.13-tool-embed-positive-selector |
terminal_positive_selector_replacement_fail_pr_closed_unmerged | Tool-specialized Tool-Embed-0.6B did not outperform same-surface BGE-M3 and missed the runtime gate. | issue #406 · PR #407 · workflow 36496824066 |
| 8 | 0.13-relative-multilingual-cross-encoder-membership |
terminal_dev_quality_and_runtime_fail_pr_closed_unmerged | Joint query-document relevance over registered, counterfactual, and background documents did not establish open-set membership and was far beyond the CPU runtime budget. | issue #408 · PR #411 · workflow 36498385690 |
| 9 | 0.13-frozen-gte-positive-selector |
terminal_positive_selector_replacement_fail_pr_closed_unmerged | Historically promising GTE retrieval did not transfer to the fresh registry surface; BGE-M3 won by 16.67 percentage points while GTE retained a good CPU runtime. | issue #409 · PR #410 · workflow 36500171432 |
| 10 | 0.13-conformal-multilingual-e5-membership |
terminal_dev_supported_recall_fail_pr_closed_unmerged | A fixed alpha=0.01 unsupported-null conformal gate reached the open-set safety/runtime targets, but the scalar E5 catalog score overlapped too strongly and vetoed 87.29% of raw-correct supported winners. | issue #412 · PR #413 · workflow 36498943508 |
| 11 | 0.13-v6h-end-to-end-operation-oos-parser |
terminal_dev_quality_fail_confirmation_unopened_pr_closed_unmerged | End-to-end multilingual encoder fine-tuning learned tool-vs-background scope well but operation semantics generalized too weakly, especially for near-domain unsupported requests. This closes the 0.13 authoritative parser/veto formulation. | issue #415 · PR #416 · workflow 36502279447 |
0.14 end-to-end agent utility¶
| # | Experiment | Decision / state | Purpose / interpretation | Evidence |
|---|---|---|---|---|
| 1 | 0.14-agent-utility-phase-a |
phase_a_passed_phase_b_authorized | Top-1 is a poor primary product metric for multi-tool capability retrieval. Top-5 preserved every required capability after the B1-v2 task-contract refreeze while schema-context ratio remained 2.383% of FULL at 250 endpoints. | issue #418 · PR #419 · freeze 36507439562 · 1c0dc93e84 |
| 2 | 0.14-b1-local-agent-ab |
running_b1_v2_protocol_validated_canonical_execution | B1 v2 corrects explicit user-argument contracts and tool-observation causality before any accepted aggregate; it remains a small-model sanity baseline. | issue #420 · PR #421 · canonical run 36529108855 · task SHA bc0b78ff... |
| 3 | 0.14-b2-strong-agent-replication |
blocked_until_strong_agent_identity_is_frozen | Strong-agent replication is required before generalizing B1 utility beyond the small local baseline. | issue #423 |
| 4 | 0.14-final-answer-quality |
blocked_until_b1_aggregate_and_b2_model_freeze | Final-answer factual quality, unit accuracy, provenance and hallucination must be evaluated separately from deterministic tool-use task success. | issue #424 |
| 5 | 0.14-corrective-state-aware-reretrieval |
no_execution_before_420_terminal | Compare static candidate exposure with execution-state-aware re-retrieval using bounded typed observations; retrieval never becomes execution authority. | issue #431 |
| 6 | 0.14-large-held-out-generalization |
freeze_sample_size_and_generation_protocol_after_b1_without_using_b1_row_failures | Create an independently frozen, materially larger multilingual held-out surface for population-level uncertainty/generalization claims. | issue #432 |
| 7 | 0.14-adaptive-shortlist-depth |
no_execution_before_b1_b2_fixed_k_evidence | Test preregistered per-query adaptive candidate depth only after fixed-K evidence; do not derive the policy from B1 errors. | issue #430 |
| 8 | 0.14-public-topk-retrieval-api |
do_not_promote_before_b1_aggregate | Stage a first-class typed Top-K public retrieval API while keeping agent orchestration and execution authority outside the core retriever. | issue #428 |
| 9 | 0.14-output-field-projection |
preregistered_dev_screen_authorized_confirmation_gated_on_b2_terminal | Isolate output-field projection, which no prior experiment separates: B1 varied how many tools were visible and #434 varied capability representation, while this holds query, exposure, route and raw response fixed and varies only what the executed tool hands back. | issue #506 · protocol |
| 10 | 0.14-output-field-projection-successor-screen |
preregistered_instrument_gate_frozen_before_any_candidate_runs | #506's screen returned a null result because its runtime emitted answer envelopes without calling a tool; the successor selects its instrument by a frozen capability gate on a separate surface rather than by looking at outcomes. | issue #510 · protocol |
Evidence-to-Action contract regression¶
Issue #1203 / PR #1204 introduced a distinct deterministic execution-boundary experiment: an eight-case frozen corpus comparing vanilla behavior, routing-only behavior, and SchemaRouter's typed evidence gate. The checked-in baseline is a regression artifact, not part of the historical routing experiment count above and not a SafeActBench reproduction. Its canonical narrative and limitations are documented in Evidence-to-Action boundary and benchmarks/evidence-to-action-v1/.