Decision routing benchmark¶

SchemaRouter includes a reproducible routing benchmark harness and a checked-in v1 corpus.

The benchmark is a measurement tool, not a claim that one provider is universally better.

Smoke run¶

python scripts/benchmark_decision_routing.py

This keeps the original three-case deterministic smoke test for packaging and CI.

Full checked-in corpus¶

python scripts/benchmark_decision_routing.py \
  --corpus benchmarks/decision-routing-v1.json

The v1 corpus contains 144 cases across:

normal routing;
near-duplicate tools/endpoints;
Korean/English multilingual queries;
long-tail phrasings;
prompt-injection-style requests;
out-of-domain requests where a bounded backend may abstain.

The corpus references only the stable benchmark registry embedded in the script and is validated in tests so unknown expected routes cannot silently enter the dataset.

v2 stress corpus and held-out splits¶

benchmarks/decision-routing-v2.json is a deterministic 1,200-case multilingual stress corpus:

16 registered routes × 60 cases = 960 in-domain cases;
240 explicit no-route / out-of-domain cases;
languages: English, Korean, Spanish, Japanese, German, and mixed Korean/English;
fixed splits: 720 dev, 240 calibration, 240 test.

The corpus is regenerated by scripts/generate_decision_routing_v2.py; tests require the generated content to match the checked-in JSON exactly. It is a template-derived stress corpus, not a substitute for independently collected human traffic.

python scripts/benchmark_decision_routing.py \
  --corpus benchmarks/decision-routing-v2.json \
  --split calibration

Tune recall width and confidence thresholds only on dev / calibration. Treat test as held out until the configuration is frozen. The v2 test split has already been consumed once for the semantic-recall checkpoint and is now a frozen regression set rather than a tuning set.

v3 untouched capability-fit holdout¶

benchmarks/decision-routing-v3.json is a separate deterministic 600-case multilingual holdout for the capability-fit experiment:

16 registered routes × 30 = 480 in-domain cases;
120 explicit no-route / out-of-domain cases;
English, Korean, Spanish, Japanese, German, and mixed Korean/English, 100 cases each;
every case is marked test;
no exact normalized query overlaps v2.

The corpus is reproducibly generated by scripts/generate_decision_routing_v3.py. Tune the capability-fit threshold only on v2 dev / calibration, freeze the chosen threshold, then evaluate v3 exactly once. After that first evaluation, v3 becomes a frozen regression set too.

Persist results¶

python scripts/benchmark_decision_routing.py \
  --corpus benchmarks/decision-routing-v1.json \
  --repeat 3 \
  --json-out artifacts/decision-benchmark.json \
  --csv-out artifacts/decision-benchmark.csv \
  --html-out artifacts/decision-benchmark.html

Useful controls:

--max-cases N for a bounded local sample;
--repeat N for repeated latency measurements;
JSON output for aggregate/report automation;
CSV output for row-level analysis;
self-contained HTML output for a portable backend/latency/abstention summary.

The HTML file contains no remote assets or scripts and escapes report metadata before rendering. It is intended for sharing one reproducible run summary.

For a dated set of benchmark JSON files, render a comparison view without rewriting the raw measurements:

python scripts/render_benchmark_history.py \
  artifacts/run-2026-09-23.json \
  artifacts/run-2026-09-30.json \
  --output artifacts/decision-benchmark-history.html

The history renderer preserves each run's timestamp, SchemaRouter version, exact source revision, corpus SHA-256, corpus path, hardware, backend, accuracy, abstention, latency, model, and device metadata. It does not normalize unlike environments, so cross-run comparisons remain valid only when measurement conditions are equivalent.

ModelQueryAnalyzer¶

Pass a provider callable using module:function syntax:

python scripts/benchmark_decision_routing.py \
  --corpus benchmarks/decision-routing-v1.json \
  --model-callable my_bench_provider:analyze

The callable receives the same structured request used by ModelQueryAnalyzer and may be sync or async. This keeps the benchmark provider-neutral.

Arbitrary bounded decision callable¶

For a typed decision model that does not expose the Jev/System One wire protocol, wrap its runtime as a normal SchemaRouter bounded decision callable and load it explicitly:

python scripts/benchmark_decision_routing.py \
  --corpus benchmarks/decision-routing-v1.json \
  --decision-callable my_decision_provider:decide \
  --decision-callable-name anyjev-local

The callable receives a DecisionRequest and returns either DecisionResult or an equivalent mapping. SchemaRouter wraps it with CallableDecisionBackend and still validates every returned option ID against the finite offered set.

This is the preferred research seam for non-wire-compatible System One-style runtimes such as experimental AnyJev/Nimble adapters. It avoids adding a permanent core integration merely to benchmark a new model family.

The callable is explicitly named trusted local Python code. SchemaRouter does not auto-discover or auto-import arbitrary installed decision providers.

Embedding backend¶

Pass a batch embedding callable using module:function syntax:

python scripts/benchmark_decision_routing.py \
  --corpus benchmarks/decision-routing-v1.json \
  --embedding-callable my_embeddings:embed_batch \
  --min-similarity 0.35 \
  --min-margin 0.05

The callable receives a list containing the query followed by one text representation per offered option, and returns one vector per input. It may be synchronous or asynchronous. SchemaRouter performs cosine ranking locally, so the benchmark can compare FastEmbed, SentenceTransformers, semantic-router encoders, or an application-specific embedding service without changing the benchmark contract.

The threshold values are workload/model specific. Do not reuse a threshold measured for a different embedding model without recalibration.

Semantic top-k recall before final decision¶

The same embedding contract can be used only for candidate recall while Laya, Jev, Ollama, or a hosted bounded backend performs final selection:

python scripts/benchmark_decision_routing.py \
  --corpus benchmarks/decision-routing-v2.json \
  --split calibration \
  --laya \
  --candidate-recall-embedding-callable benchmarks.multilingual_embedder:embed \
  --candidate-recall-limit 4

Controls:

--candidate-recall-embedding-callable module:function;
--candidate-recall-limit N;
--candidate-recall-min-similarity FLOAT;
--candidate-recall-min-margin FLOAT.

The checked-in research callable uses sentence-transformers/paraphrase-multilingual-MiniLM-L12-v2 only in benchmark environments; sentence-transformers is not a SchemaRouter package dependency.

The Research Benchmark compares recall widths on the calibration split first and sweeps confidence thresholds offline. The held-out test split is evaluated only after a configuration is selected.

Generic System One-compatible provider¶

Use the model-neutral System One backend when a hosted or self-hosted runtime exposes the Jev-compatible /v1/systemone contract. This lets the same benchmark compare Kev or future compatible decision models without adding another backend implementation.

Install the neutral extra:

pip install -e ".[systemone]"

For a local Kev-style server:

export SYSTEM_ONE_API_KEY="local"
python scripts/benchmark_decision_routing.py \
  --corpus benchmarks/decision-routing-v1.json \
  --system-one-base-url http://127.0.0.1:8009 \
  --system-one-provider kev-local \
  --system-one-model kev-latest \
  --system-one-min-confidence 0.70

Available controls:

--system-one-base-url URL enables the generic compatible backend;
--system-one-model MODEL pins the provider model/checkpoint alias;
--system-one-provider NAME records a stable provider label in rows and reports;
--system-one-timeout SECONDS;
--system-one-min-confidence FLOAT;
--system-one-api-key-env NAME selects the environment variable containing credentials.

Credential values are never written to benchmark output. Reports record only whether a key was configured, plus the non-secret provider/model/base-URL/runtime configuration required for reproducibility.

Wire compatibility is not a quality claim. When changing model/checkpoint/runtime, recalibrate confidence only on development/calibration evidence and evaluate the frozen configuration on an untouched holdout.

Jev¶

Install the optional integration and configure TypeSafe locally:

pip install -e ".[jev]"
export TYPESAFE_API_KEY="..."
python scripts/benchmark_decision_routing.py \
  --corpus benchmarks/decision-routing-v1.json \
  --jev

Optional controls include --jev-model, --min-confidence, --input-cost-per-million, and --output-cost-per-million.

Provider pricing is never hard-coded because it can change independently of SchemaRouter.

Laya¶

Install the optional local decision runtime:

pip install -e ".[laya]"

Run Laya against the same checked-in corpus:

python scripts/benchmark_decision_routing.py \
  --corpus benchmarks/decision-routing-v1.json \
  --laya

By default, Laya uses its local router to choose an English or multilingual checkpoint from the bounded request state. Optional controls include:

--laya-model english|multilingual|typed-decisions to pin a checkpoint;
--laya-device cpu|cuda|mps to pin the trusted local device;
--laya-preload to preload checkpoints before measurement;
--laya-max-loaded N to control resident checkpoint count;
--laya-min-confidence FLOAT to measure confidence-gated abstention;
--hardware-label TEXT to attach the concrete machine/GPU description to the report;
--source-revision TEXT to record the exact code revision when not supplied by CI;
--decision-recall-on-empty to explicitly let an enabled bounded decision backend inspect the registered endpoint catalog when lexical candidate recall is empty;
--candidate-abstention inherit|deterministic|no_route|error to separate explicit backend abstention from provider-error fallback behavior.

Laya rows record the routed checkpoint plus requested_device, decision confidence, and, when Laya exposes the loaded agent device, actual_device: an unavailable CUDA/MPS target can fall back to CPU and should not be counted as an accelerator result.

Empty-candidate recall is off by default. When enabled, SchemaRouter does not invent a route: it exposes only endpoints already registered in the trusted local catalog. If the backend errors or abstains after this expansion, planning fails closed with no arbitrary deterministic route because there was no lexical candidate to fall back to. For workloads that need model-driven no-route behavior, combine confidence gating with candidate_abstention="no_route" and calibrate the threshold on representative data.

A zero-threshold run can be recalibrated offline without repeating model inference:

python scripts/calibrate_decision_threshold.py \
  artifacts/laya/recall-on-empty/report.json \
  --backend laya:auto \
  --json-out artifacts/laya/calibration.json \
  --csv-out artifacts/laya/calibration.csv \
  --html-out artifacts/laya/calibration.html

The calibration can replay low-confidence cases either through the recorded deterministic fallback route or as a final no-route result with --abstention-mode no_route. It reports overall/category accuracy, confidence coverage, backend abstention, final no-route rate, expected no-route recall, and expanded-candidate selection rate for thresholds 0.00 through 0.95. This separates threshold tuning from model/runtime latency and avoids paying for repeated inference.

Benchmark JSON also records the installed Laya, PyTorch, and Transformers package versions when Laya is enabled so dated results can be reproduced against the actual local runtime.

For mixed-language latency measurements, distinguish cold-swap from steady-state resident conditions. With auto language routing, max_loaded=1 can repeatedly evict English and multilingual checkpoints as the corpus alternates languages. The checked-in Research Benchmark therefore uses --laya-max-loaded 2 --laya-preload: both language checkpoints are loaded before the timed per-case loop and remain resident when memory allows. Use max_loaded=1 only when measuring constrained-memory checkpoint churn.

Published results should record the exact Laya package version, checkpoint/routing policy, hardware, device, preload policy, confidence threshold, source revision, corpus SHA-256, and repeated-run count. The benchmark report records these reproducibility fields directly when available. Do not compare upstream/provider benchmark numbers directly unless the prompts, options, corpus, and measurement conditions are equivalent.

Ollama¶

Run an already-installed local Ollama model against the same corpus:

python scripts/benchmark_decision_routing.py \
  --corpus benchmarks/decision-routing-v1.json \
  --ollama-model your-installed-model

The default API endpoint is http://127.0.0.1:11434. Use --ollama-base-url for another trusted endpoint and --ollama-timeout to adjust the per-request timeout. Server-supported local runtime options can be passed without hard-coding vendor/version-specific knobs:

python scripts/benchmark_decision_routing.py \
  --ollama-model your-installed-model \
  --ollama-options-json '{"your_runtime_option": 1}' \
  --hardware-label "RTX workstation"

SchemaRouter records the options object in the benchmark report. It does not infer the actual Ollama CPU/GPU placement because that decision belongs to the Ollama server/runtime.

The benchmark records Ollama prompt/evaluation token counters when the server reports them. A local model result should also record the exact model tag, quantization/runtime configuration, hardware, and Ollama version when publishing evidence; model tags alone are not enough to reproduce local latency or quality.

Provider pricing is irrelevant for a purely local Ollama run unless the operator explicitly assigns a local cost model.

Metrics¶

Each row records:

case ID, category, fixed split, and language label;
provider/model identifier when reported;
requested and actual local device when the backend can report them;
expected and predicted tool.endpoint;
correctness;
invalid-plan state;
end-to-end planning latency;
whether the bounded decision backend was actually invoked;
whether empty lexical recall was explicitly expanded to the registered catalog;
bounded-backend abstention and deterministic fallback state;
input/output tokens when reported;
optional cost estimate;
provider/planner error.

The aggregate report includes:

final-plan routing accuracy (an abstention case is correct only when the final plan has no route) with a 95% Wilson score interval;
invalid-plan rate;
error count;
bounded-backend invocation count/rate;
final expected no-route recall with a 95% Wilson score interval;
abstention rate;
expected-abstention recall for bounded backends, reported separately from final-plan accuracy;
fallback count;
mean, p50, and p95 latency;
category-level accuracy with per-category 95% Wilson score intervals;
split-level and language-level accuracy;
token totals and optional estimated cost.

Offline versus live evidence¶

The corpus and deterministic smoke path are suitable for CI because they require no external provider.

A live provider benchmark is kept separate. Network availability, model revisions, credentials, rate limits, provider-side changes, local accelerator availability, memory, and runtime placement are external variables. Publish live results with the exact date, model identifier, configuration, corpus revision, and repeated-run count.

The reported 95% Wilson intervals quantify binomial sampling uncertainty for the checked-in corpus. They do not account for model/provider drift, correlated repeated cases, hardware variance, or distribution shift. Do not infer provider superiority from the three-case smoke set or from a single live run.

Operation-capability fit protocol¶

Near-domain unsupported operations are evaluated separately from ordinary out-of-domain traffic. The v5 operation corpus contains 576 multilingual cases: 288 supported route operations and 288 near-domain unsupported operations, split into 384 development and 192 calibration cases.

The operation-fit threshold must be selected from v5 development/calibration results only. The 600-case v6 operation holdout contains 384 supported route cases, 192 near-domain unsupported-operation negatives, and 24 ordinary out-of-domain negatives, balanced to 100 cases per language group. v6 was already evaluated with the earlier operation-fit input representation and is therefore regression evidence only; it must not be reused for a new untouched generalization claim.

The protected GitHub Actions holdout job now targets v9 and remains manual-only. It is skipped for pull-request runs and ordinary manual benchmark runs. v6 and v7 are consumed regression evidence; v8 was accidentally consumed by a calibration diagnostic path and therefore cannot support an untouched claim. v9 was reserved before the alias-aware operation-fit implementation was evaluated. After the alias-aware v5 threshold was frozen at 0.40, v9 was consumed exactly once on 2026-09-26. The permanent workflow guard remains manual/threshold-gated so the consumed corpus cannot be silently presented as a fresh holdout.

The frozen upstream stack for this experiment is:

semantic candidate recall top-k = 2;
broad capability-fit min similarity = 0.25;
same-tool endpoint disambiguation min margin = 0.03.

Only the operation-fit threshold is varied during v5 calibration. For the label-cleaned operation-fit surface, freeze that threshold before evaluating v7 and do not retune it from v7 results.

For the label-cleaned surface, the v5 sweep over 0.05 through 0.70 froze operation_fit_min_similarity = 0.45 before v7 evaluation. The selection rule is maximum calibration accuracy, with dev accuracy used as a stability check. At 0.45, calibration accuracy was 65.625% and dev accuracy was 66.667%; adjacent thresholds 0.40 and 0.50 reached 60.938% and 60.417% calibration accuracy respectively. This frozen value must not be changed based on v7.

The one-shot v7 evaluation at the frozen 0.45 threshold produced 46.833% overall accuracy (95% Wilson CI 42.873%–50.834%), 28.646% routed accuracy on supported operations (24.353%–33.362%), 76.562% accuracy on near-domain unsupported operations (70.086%–81.997%), and 100% accuracy on ordinary out-of-domain cases. This is treated as a distribution-shift diagnostic, not as a target for threshold retuning. The v7 holdout is now consumed and the workflow guard has been restored to manual-only.

For the later trusted-alias surface, the post-alias v5 sweep froze operation_fit_min_similarity = 0.40 using development/calibration only. The one-shot v9 evaluation at that frozen threshold produced 51.167% overall accuracy (95% Wilson CI 47.172%–55.146%), 38.281% routed accuracy on supported alias-operation cases (33.558%–43.236%), 70.833% accuracy on near-domain unsupported operations (64.046%–76.804%), and 100% accuracy on ordinary out-of-domain cases (86.202%–100%). Language-group accuracy ranged from 40% (Korean) to 57% (Japanese). The result is retained as untouched generalization evidence: aliases improved the supported-route surface relative to the consumed v7 diagnostic, but operation selection recall remains materially weaker than unsupported-operation rejection. v9 is now consumed and must not be used for threshold retuning.

Post-v9 concise surface and v10 generalization holdout¶

After the v9 one-shot result, stage-attributed v5 dev/calibration profiling showed that operation-fit abstention remained the dominant supported-route suppression stage. Three representation variants were compared at the already-frozen 0.40 threshold using v5 development/calibration only:

concise endpoint name + trusted aliases + endpoint description: 69.792% balanced operation score on both dev and calibration after endpoint disambiguation;
repeated natural-language Supported operation: phrases: 64.062% dev / 63.542% calibration;
independent max-over-alias variants: 59.635% dev / 60.417% calibration.

The concise representation was selected before inspecting v10. A fresh threshold sweep from 0.05 through 0.70 then used v5 dev/calibration only. The robustness score was the lower of the dev and calibration balanced operation scores. The selected threshold remained operation_fit_min_similarity = 0.40: 0.35 reached 67.188% robustness, 0.40 reached 69.792%, and 0.45 reached 67.708%.

benchmarks/decision-routing-v10-operation-generalization-holdout.json is a 600-case, six-language test-only generalization corpus frozen before this representation optimization. It contains 384 supported operations, 192 near-domain unsupported operations, and 24 ordinary out-of-domain requests.

With the representation and threshold frozen, v10 was evaluated exactly once on 2026-09-26 using semantic recall top-k 2, capability-fit 0.25, operation-fit 0.40, and endpoint-disambiguation margin 0.03. The final stack produced:

55.667% overall accuracy (95% Wilson CI 51.668%–59.593%);
42.188% supported-operation routed accuracy (37.349%–47.181%);
77.083% near-domain unsupported-operation rejection (70.642%–82.462%);
100% ordinary out-of-domain rejection (86.202%–100%);
0 provider/planner errors and 0 invalid plans;
language-group accuracy from 47% (Korean) to 61% (Japanese).

The final v10 error taxonomy contained 199 missed routes, 44 false routes, and 23 wrong-endpoint routes. Of the missed routes, 162 were attributed to operation-fit and 37 to capability-fit. This is retained as a limitation signal, not a reason to retune against v10. Any further operation-fit optimization must reserve a new untouched holdout before implementation; v10 is now consumed regression evidence.

v11 pairwise operation-fit generalization¶

After v10, threshold/margin variants and replacement bi-encoders were evaluated using v5 development/calibration only. Neither global margins, two-tier rescue, multilingual MPNet, nor multilingual E5 improved the pre-registered robustness objective over the concise multilingual MiniLM bi-encoder. A pairwise reranking experiment was the first alternative scoring form to exceed that objective on v5 dev/calibration: BAAI/bge-reranker-v2-m3 with sigmoid-normalized pair logits and min_score = 0.01 reached 71.354% robustness, compared with 69.792% for the MiniLM bi-encoder.

Before that pairwise result was inspected, v11 had already been reserved as a fresh 600-case, six-language test-only holdout containing 384 supported operations, 192 near-domain unsupported operations, and 24 ordinary out-of-domain requests. The frozen BGE candidate was evaluated once on 2026-09-26 at source revision 57af2fd18d254fa1a9cd5855a158ae74b94adfc7 with corpus SHA-256 efa8cd371bc7613895e44a915d78c77f5b88239c64a8de52d9c407839660d816. The artifact benchmark-operation-fit-v11-heldout-36239340782 has SHA-256 f977511c7d65bc94e2dba9142b6cedf378ceaae1b8418c351e85b550f191dfd1.

The final BGE stack produced:

51.667% overall accuracy;
25.781% supported-operation routed accuracy;
97.396% near-domain unsupported-operation rejection;
100% ordinary out-of-domain rejection;
61.589% balanced operation score;
0 provider/planner errors and 0 invalid plans;
277 missed routes, of which 251 were attributed to operation-fit.

Because the BGE result traded substantial supported recall for stronger rejection, the already-frozen MiniLM 0.40 configuration was then run once on the already consumed v11 corpus strictly as a paired diagnostic, not for tuning. That baseline produced 54.833% overall accuracy, 40.625% supported-operation routed accuracy, 77.604% near-domain rejection, 100% ordinary OOD rejection, and a 59.115% balanced operation score. Its diagnostic artifact benchmark-operation-fit-v11-baseline-diagnostic-36240431389 has SHA-256 1048405ab34b41371004247e15934f5086a53977a14d4957bd14c6d69e194dee.

Thus the pairwise candidate generalized a +2.474 percentage-point improvement on the pre-registered balanced objective, but not an overall-accuracy or supported-recall improvement. SchemaRouter therefore exposes pairwise scoring as an optional bounded backend rather than selecting BGE as a library default. v11 is consumed evidence and must not be used to select any later hybrid rule, threshold, representation, or model.

Post-change operation holdout (v7)¶

benchmarks/decision-routing-v7-operation-post-change-holdout.json is a 600-case, six-language test-only corpus reserved after removing tool-domain labels from the bounded operation-fit embedding surface. It contains 384 supported operations, 192 near-domain unsupported operations, and 24 ordinary out-of-domain requests. Its normalized queries are checked to be disjoint from v2 through v6.

The corpus was added before measuring the label-cleaned operation-fit implementation. Do not use v7 to select the operation-fit threshold. Calibrate only on v5; after freezing the threshold, evaluate v7 once for the post-change generalization claim. Repeated v6 runs are regression evidence only because v6 had already been consumed before the operation-fit input representation changed.

Reserved operation-alias holdout (v8)¶

benchmarks/decision-routing-v8-operation-alias-holdout.json is a fresh 600-case, six-language holdout reserved before adding trusted endpoint operation aliases. It contains 384 supported-operation cases, 192 near-domain unsupported-operation cases, and 24 ordinary out-of-domain cases. Its normalized queries are disjoint from v2 through v7.

v8 was reserved before alias implementation, but an early alias-aware calibration workflow accidentally referenced v8 from the per-threshold diagnostic step. That exposed v8 across the threshold sweep, so v8 is now treated as consumed diagnostic data and must not support an untouched generalization claim. The workflow has been corrected to use the already-consumed v7 diagnostic corpus during threshold selection.

Alias-aware threshold selection rule¶

After adding trusted EndpointSpec.operation_aliases, operation-fit threshold selection uses only v5 and the already-consumed v7 diagnostic corpus. For each threshold, compute a balanced operation score for each corpus as the arithmetic mean of supported-operation accuracy and near-domain unsupported-operation accuracy. The primary robustness score is the lower of the v5 calibration balanced score and the v7 diagnostic balanced score.

Freeze the threshold that maximizes this robustness score. Use v5 development balanced accuracy as the first tie-breaker, then prefer the lower threshold as the second tie-breaker to preserve supported-request recall. Neither consumed v8 nor reserved v9 may participate in threshold selection or alias design. After the alias vocabulary and threshold are frozen from v5 plus the already-consumed v7 diagnostic, v9 is evaluated once.

Reserved alias-aware holdout (v9)¶

benchmarks/decision-routing-v9-operation-alias-holdout.json is the replacement untouched 600-case, six-language holdout reserved after discovering the v8 workflow contamination and before inspecting alias-aware calibration results. It contains 384 supported operations, 192 near-domain unsupported operations, and 24 ordinary out-of-domain requests. Its normalized queries are disjoint from v2 through v8.

v9 is manual-only and threshold-gated in GitHub Actions. Do not run it until the alias vocabulary and operation-fit threshold are frozen using v5 and the already-consumed v7 diagnostic corpus.