Research program and issue triage (2026-10-19)

Research navigation: Knowledge index for agents and maintainers — topic → evidence → implementation → issue, without duplicating the experiment ledger.

Status: maintained research strategy / exposed evidence; not a new benchmark, release approval, or independent performance result. This is the Git-tracked bilingual research wiki, rather than a separate GitHub Wiki, so every change is reviewable and reproducible. Baseline source audit: main a6ec24cf3fa1a4c0c0869e3853ab65d5fda1db83 on 2026-10-19. G5 opt-in integration evidence updated 2026-10-22 on a dedicated implementation branch (not a release/default promotion).

1. Mission and scope

Primary research question: Can a source-attested, typed, authorization-scoped schema graph expose only the necessary Tool/Endpoint/Parameter/ResponseField capabilities to an external agent, at bounded K and schema/context cost, without sacrificing full task completion, correct no-fit abstention, or execution safety compared with a full-catalog baseline?

The boundary is capability discovery and governed typed execution, not a new autonomous agent, LLM gateway, translation model, benchmark oracle, graph database, or tool-invoker replacement. Keep the one canonical registered catalog across OpenAPI, MCP, OPTIMADE, Python and other native sources. Query-concept-to-FieldGroup matching, producer-to-consumer edges and source/permission proofs may narrow candidates, never create permission. External host/agent retains actual model behavior and call proposal; trusted execution revalidates authority, arguments, source and observations.

The original SchemaRouter paper reports a 110-question domain study, not a guarantee on unseen heterogeneous 5k-tool catalogs or downstream agents. The frozen independent 0.14 experiment is an important negative: FULL 49.62%, SR-5 29.19%, SR-10 31.03%, SR-PROGRESSIVE 25.09%, ORACLE 70.98% task success (Issue #432, failure analysis #1339). Never rewrite that outcome when later exposed development scores improve.

2026-10-22 integrated G5 opt-in slice: implemented boundary vs E1 blocker

Product implementation (opt-in): schemarouter.retrieve_authorized_concept_graph composes the existing owner-declared resolve_native_concepts → complete authorized FieldGroup retrieval → exact-version NativeSchemaGraphIndex.lookup. It returns immutable NativeConceptGraphSelection with route_sets, native tool/endpoint fingerprint triples, declared semantic output IDs and all proved original producer-field→consumer-parameter bindings. A complete input-free direct endpoint is preferred. An endpoint whose required inputs are not satisfied is not promoted to an input-free direct route; it must be supported by the existing complete typed one-hop join. Both are discovery-only: executable_authority=False.

The exact opt-in wiring, in an explicitly principal-authorized host with already reviewed source metadata, is:

from schemarouter import (
    NativeSchemaGraphIndex,
    NativeSchemaPostingIndex,
    retrieve_authorized_concept_graph,
)

# router: existing SchemaRouter with explicit authorization_policy and native source tools
# catalog: owner-reviewed NativeConceptCatalog with EN/KO/JA aliases and exact typed fields
# principal: existing PrincipalContext scoped to this request
index = NativeSchemaGraphIndex.build(NativeSchemaPostingIndex.build(router.registry))
result = retrieve_authorized_concept_graph(
    router, "문의 상태", principal=principal, catalog=catalog, index=index, k=5,
)
if result.status == "complete":
    print(result.route_ids)  # Authorised native *route IDs*, not a tool-call grant

This example presumes the original host has genuinely declared "문의 상태" in the catalog; SchemaRouter does not translate or infer it. Equivalent owner-declared EN/KO/JA aliases resolve to the same typed constraints. no_proven_hit reveals no hidden identities; if more than five distinct route IDs are needed by complete alternatives without verified ranking, over_k_unranked returns no routes, never an arbitrary Top-5. The graph index, registry revision, policy/data scope, availability, native fingerprints and catalog contents are rechecked, and neither queries nor discovery results authorize invocation. This first slice supports direct + one-hop, not automatic multi-hop orchestration.

Agent integration: AuthorizedConceptGraphFixed5Selector extends the existing callback in scripts/research_authorized_concept_fixed5_agent_bridge.py and plugs into the pre-existing EVAL-FIXED5-OPTIN generated-agent path. Only AuthorizedAgentToolSurface definitions scoped to the principal, original query and exact source revision are shown. The selector verifies live policy content, hidden/ref-safe field projections, all selected source fingerprints, K5 and actual serialized tool-definition byte budget. The previous concept selector and ordinary retrieval defaults remain unchanged. Controlled E0 integration tests showed that an agent using the existing loop can see both producer and consumer routes and complete two calls, but those model-stub/fixture outcomes are not native E1 or independent E2 evidence.

Native E1 preflight: scripts.research_real_qwen_open_meteo_two_hop_e1 has an explicitly named concept_graph_optin and compare_graph_native_e1 comparison with all_visible and lexical_k5. It reuses the original Open-Meteo source invokers, source-state checks and strict final factual scorer. Run source qualification only (without loading model weights or calling the provider):

PYTHONPATH=src:. python -m scripts.research_real_qwen_open_meteo_two_hop_e1 \
  --compare-graph --out /tmp/schemarouter-native-g5-e1.json

For a separately approved original-source intake, the same CLI can be given --host-catalog /path/to/reviewed-concepts.json and repeated --approved-host-root tool.endpoint arguments. These values are the host's reviewed declarations, not inferred from E1 queries or evaluation gold. The E1 module revalidates the live original source/policy/typed graph before any call, and loads pinned model weights offline only after this qualification passes. Passing flags alone never cures an unattested source.

Observed 2026-10-22 result: BLOCKED_SOURCE_SEMANTICS, task_success_rate: null, 0 provider HTTP attempts, 0 model turns (qualification preflight). The original build_live_router exposes source type but currently lacks a license declaration, exact scalar output semantic IDs, typed producer-coordinate→weather-input G5 edges, an explicit principal authorization policy and qualified host-reviewed source-root input contract. Those are real source/host intake prerequisites; inserting inferred fields/permissions simply to improve an E1 score is prohibited. Verify upstream Open-Meteo usage terms and exact source schemas separately before any permitted live paired test. The model's pinned local weights were not loaded and therefore no native agent task outcome percentage exists for the new G5 arm. Even when qualified, this source exposes only two operations, so K5 cannot by itself demonstrate schema-cost reduction versus FULL; it tests correct two-step outcomes first.

2026-10-23 optional documented source intake (research only): Official Geocoding and Forecast field contracts have been cross-checked separately from task answers. The unmodified default above remains unqualified; the explicit --documented-noncommercial-e1 mode pins the two original adapter fingerprints, declares the WGS84 latitude and longitude scalar semantics and a matching forecast consumer, a deny-by-default two-route reader policy, a separately declared EN/KO/JA concept catalog, and one approved parameterized geocoding root. Only this research harness projects bounded native arguments to strict G5 scalar types, while the bound host invoker independently enforces the original enum, length and coordinate-range rules before any HTTP; the original adapter and API host stay unchanged. Unit tests with scripted model and httpx.MockTransport verified all three typed joins, exact two-step tool execution, final source grounding and rejected invalid preflight arguments. A model-free documented-source preflight reported BLOCKED_MODEL_RUNTIME (no provider calls). A subsequent actual pinned-Qwen/Open-Meteo paired run recorded nine source episodes, including one strict completion, but provider failures made the comparison INCOMPARABLE_SOURCE_FAILURE with task_success_rate: null; it proves neither a comparative G5 win nor independent E2. See the retained source-native run record. The new flag is an opt-in for Open-Meteo's free non-commercial conditions, not a commercial licence, human signature or E2 review. Commercial SchemaRouter hosts must independently use a permitted customer API/contract; the library's MIT licence does not change Open-Meteo API restrictions.

PYTHONPATH=src:. python -m scripts.research_real_qwen_open_meteo_two_hop_e1 \
  --compare-graph --documented-noncommercial-e1 \
  --out /tmp/schemarouter-native-g5-documented-e1.json

Sequencing: obtain legitimate source-owner reviewed semantic/permission metadata and model availability, then rerun genuine native paired EN/KO/JA E1; record source/observation drift as incomparable and source/provider errors separately from model-final mistakes. Independent publisher/task-author-disjoint E2 remains BLOCKED_EXTERNAL under #1346; #1535, #1560 and #1477 remain open. There is no default behavior change or permission to merge #1356. Live milestones: #1700 → M1 (only measured integrated G5 bottlenecks), #1711 → M5 (CodeGraph nonblocking), global tracker #200 intentionally unassigned. See live addendum; the initial 30-item manifest is a protected historical initializer.

2. Research architecture and non-negotiable invariants

1. Ground truth of capability: registered source-native schema, exact endpoint fingerprints, reviewer-approved semantic IDs, units, license, source version, and typed input/output constraints. Unreviewed descriptions and LLM-extracted intent confer no new authority.
2. Discover: independently resolve query concepts to conjunctive FieldGroups; compute complete authorized and data-scope-visible candidates before K. Use the original capability_graph.py typed compatibility graph and existing G5 index, not a second graph/router. No arbitrary Top-5 cut for >5 equally verified graph paths: report over_k_unranked pending valid ranking.
3. Fit and abstain: route recall, parameter compatibility, field-set completeness and independently calibrated NO_FIT are different endpoints. score > 0 and semantic similarity are not authorization or fitness probability.
4. Execute: external agent chooses a route and arguments. Trusted host independently validates principal, visibility, snapshot, call schema, destructive policy, executed observations and result provenance. A candidate reference is not a valid execution receipt.
5. Measure: registered constraints, safety, required route/field/operation recall, unsupported false admissions, supported false refusals, real final task pass, schema tokens and CPU/RSS. No unseen labels or expected next tool/arguments may enter retrieval or model prompts.

3. Empirical status and priority order

Priority	Workstream and owner	Evidence now	Next admissible gate
P0	Source identity, security, field privacy: #1450, #1532, #1269	Code fixes and synthetic negatives exist; nested/ref leakage and independently justified action predicates still require review	Zero hidden metadata or unauthorized destructive actions; drift and adversarial contract tests before promotion
P1	Canonical full-catalog G5 typed graph: #1560, #1530	Opt-in one-hop G5, 5k synthetic CPU and 0/1/5/6-role tests merged; no validated product default ranking or E2	Independent source-authored fields and complete graph-path Recall@5 under fixed K, permission and schema budgets
P1	Multilingual cross-language retrieval: #1339, #1473, #1403	Frozen 500-tool initial required-route recall: EN 153/160, KO 0/160, JA 0/160. Lexical Unicode fixes alone cannot translate English-only catalogs	Query-disjoint EN/KO/JA source-owned tasks and OOD hard negatives; measure field and route recall before any default change
P1	No-fit and ranking: #1582, #1444	Public BFCL V4 LIVE baseline 8/16 supported accepts, 144/880 irrelevant false accepts; stricter familywise diagnostic 4/16, 91/880. Both insufficient	Pin/licence CPU reranker comparison on disclosed E1 only, then independent calibration/confirmation on untouched source
P1	End-to-end agent value: #1477, #1348	Real Qwen3/Open-Meteo disclosed three-language E1 repeatedly 0–1/3 strict tasks; host-source failures and early model finals are separate	Reuse existing FrozenCorpusExecutor; independent task-paired FULL / lexical SR5 / typed G5 / hybrid / no-fit, and grounded final-answer metrics
P2	External validation and release: #1346, #1345, #1344	ClicShopping, SafeAct, CYT, Xerrion, Laya/pi-jev have developer/preflight work; publisher-independent required inputs remain partly BLOCKED_EXTERNAL	Finish executable/source-permitted gates, explicit blocked ledger, protected release checks, then v0.18; post-release comparisons last

Consequence: prioritize a valid no-fit and end-to-end benchmark over additional exposed two-hop Qwen reminder prompts. Existing Draft #1580 and #1584 are diagnostic only, not product improvements. Do not merge held release PR #1356 or change original 0.14 scores.

4. Comparative protocol and falsification

Use a single pinned native source catalog, identities, principal, K=5, schema-byte/token budget, model, decoding seed, host invoker, and task split. Compare FULL and ordinary lexical SR-5 to opt-in semantic SR-5, typed concept/FieldGroup G5 SR-5, and hybrid + independently calibrated disposition. ORACLE is diagnostics only. Do not substitute a synthetic executor for live provider evidence; freeze an independent author-owned cohort before outcome inspection.

Measurement	Unit / denominator	Failure means
Route / operation / required field-group recall	Supported query and all required routes/fields; report complete-set coverage separately	A seemingly correct Top-1 can still omit a required second hop or field
Supported false refusal vs unsupported false admission	Separate supported and irrelevant/OOD/forbidden counts; include empty menu cases separately	Lower false admission at cost of severe positive recall loss is not an unqualified win
Task and final-grounding success	Paired independent task, fully executed observed output; failure taxonomy by stage	A provider response or CI-green run is not successful agent completion
Safety and provenance	Count unauthorized action attempts and successful unauthorized effects, hidden fields, drift, original source validation	Any unauthorized destructive execution or hidden-data disclosure is a blocking failure
Efficiency	Actual agent-visible schema bytes/tokens, index build, warm CPU p50/p95/p99, RSS, end-to-end elapsed	Single synthetic GitHub runner timings are not a commercial SLO

Preregister primary task-level success and safety gates; treat changes in route recall, score, or tokens as secondary until their causal effect is measured. Include paired uncertainty and language/catalog strata; fixed thresholds, valid controls, and no hidden-gold route refill. Failed E2 gates remain VALID_NEGATIVE, not an opportunity to repair using consumed test rows. Distinguish E0 synthetic mechanism, E1 source-exposed development, and E2 independently authored holdout. Source network timeouts are SOURCE_FAILURE, not model misrouting. Abstaining above K and uncalibrated no-fit both require explicit denominators.

The existing A10 no-fit diagnostic leaves plan_within_declared_limits and candidate_within_declared_limits null (unresolved) while any supported or unsupported case still needs arguments or an observation. Counts, error rates and denominators remain unchanged; a valid clarification or ambiguity in the other strata does not block this binary diagnostic. Null is never a pass, and a true E0 diagnostic still requires separately observed task completion and independent review before an E2 qualification.

The new A7 E0 opt-in captures the complete principal-visible catalog with capture_authorized_agent_tool_surface(..., experimental_full_catalog=True) and compares FULL with public prepared field-session K5 using run_paper_paired_agent_e0(..., experimental_full_final_oracle=True) (also available in its counterbalanced wrapper). Both arms use fresh matching agents and the same final-answer prompt. FULL retains source/byte limits and supports the native KeywordAnalyzer; existing K5 and frozen conditions are unchanged.

Existing passed remains an execution metric. Private original #424-format gold enables separate execution_and_structured_final_passed: execution and exact required values, units and provenance must pass. Numeric checks preserve large integer precision. Gold or usage absence stays null; missing usage makes all_model_usage_complete=False. Only allowlisted outcomes leave the private oracle, without final text or gold. Free-text semantic truth is not verified, and neither these E0 observations nor the flags establish independent E2, release approval or default promotion.

5. Prior art: what changes our next experiment

Work and authoritative source	Relevant finding	Concrete transfer / limitation
SchemaRouter (Cho, 2026)	Source-typed schema graph, field projection and provenance with domain-level efficiency gains	Retain original field-aware graph thesis; prior 110-task outcomes do not establish current multi-provider agent generalization
ToolRet (Shi et al., ACL 2025)	Heterogeneous benchmark, 7.6k retrieval tasks / 43k tools; conventional IR can underperform on tools	Benchmark retrieval fidelity at source-field and task level rather than trust generic embedding scores; verify license/availability before intake
ToolRerank (Zheng et al., LREC-COLING 2024)	Hierarchy-aware reranking and adaptive candidate truncation for multi/single tool queries	Candidate for opt-in bounded rerank after source-authorized eligibility; ranking is not authority or safe abstention
BFCL V4 (Berkeley)	Function relevance/irrelevance, live and multi-turn/agentic/format categories	Keep original source labels and positive-vs-negative confusion counts; public LIVE evaluated repeatedly is E1, not unseen E2
ToolSandbox (Lu et al., 2024)	Stateful tool execution, implicit dependencies and intermediate milestones	Pair field/route exposure with actual multi-step execution, state and task-final success
FAIL-TaLMs (Treviño et al., NAACL 2025)	Missing tools and underspecified requests challenge tool agents	Distinguish unknown capability, missing arguments, permissions and provider-unavailable from semantic irrelevance
SafeActBench (Lin et al., 2026)	Correct final effect does not prove evidence justified a consequential action	Use case-blind policy predicates, time-ordered evidence/argument links, and independently reviewed public action contracts; 393 genuine V1 model trajectories remain unrun
EDGE / KOPA-Bench (Kim et al., 2026 preprint)	145 Korean public API multi-step tasks and live-execution-validated output→input dependency graph	Closest methodological analog for G5 typed joins; maintainer repository still lists task data, 1,781 EDGE trajectories and model weights as coming soon as of audit. Do not report an unrun KOPA evaluation or self-synthesize independent gold
ACEBench (Chen et al., EMNLP 2025)	Normal, ambiguous/incomplete special, and interactive agent tool-use strata	Preregister ambiguity, absent parameters, unsupported no-fit and operation fit separately; compare true model/host outcomes, not only tool-name ranking
tau2-bench (Sierra Research)	Task success within policy-constrained, stateful simulated customer-service conversations	Borrow task-level, state and policy violation rubrics; simulation is not independently verified real OpenAPI source execution
AgentDojo (Debenedetti et al., NeurIPS 2024)	Tool-response prompt injection attack / defense benchmark with task-success controls	Adversarially inject lower-trust tool observations and measure authorized action + legitimate completion jointly; never let source content rewrite policy
CaMeL (Debenedetti et al., 2025)	Trusted control/data-flow enforcement with capability-limited external effects	Separate query instructions from untrusted observation and validate dataflow/authorization; a research security architecture, not plug-in performance evidence for SchemaRouter
International Tool Calling (Zhang & Zhu, 2026 preprint)	Multilingual/international API task evaluation; geography and language matter	Evaluate natural original KO/JA/EN queries against English-only source schema; review benchmark provenance/license first
MCP 2026-07-28 specification update	Authorization hardening and cache metadata for enumerated tool/resources	Scope/version/cache eligibility to principal and authenticated source; a cached tool menu cannot itself grant invocation authority

Gap after this review: an independent bilingual/multilingual source-authored tool-and-field-level, multi-hop, OOD/forbidden task set suitable for fixed-K paired agent utility is still the central blocker, rather than a new public model score. Do not generate synthetic labels and call them publisher gold.

6. Issue lifecycle and minimal portfolio

Every open issue must state owner workstream, one independently checkable deliverable, evidence level, terminal decision, and link to one canonical parent. One PR/experiment may end negatively: close its narrow issue with completed when verified implementation/evidence exists, or not_planned when the prospective method is empirically rejected; never imply broad E2/production completion. Archive the exact head, Actions run, artifact and negative denominators. Keep broad external, security and release gates open until their own acceptance is met.

Close narrow completed scopes: #1538 G1, #1540 G2, #1551 concept composition, #1558 G4, #1562 G5 index, #1566 dense CPU study, #1572 state-ready E1, #1574 opt-in GET retry E1, #1452 exact-field read-only scan lower-bound with forged-host security counterexample, #1473 fixed-K scoped multilingual prototype and source-provider E1, and #1444 conjunctive original OpenAPI enum DEV gate. All are narrow E0/E1 evidence, not independent accuracy or default promotion. Close negative/deferred experiments: #1576 familywise false-fit correction #1579 proactive model-progress reminder, and #1578 generic completion feedback (PR #1580 closed unmerged after E1 without proven benefit). Their evidence remains linked. Parent #1560 and #1477 stay open.

Do not close merely due to PR merge: #1532 nested/ref scope-leak residuals, #1535 independent required-field coverage, #1545 protocol-disjoint validation, #1583 actual final JSON diagnostic live mismatch cohort. The separately proposed #1585 prompt variant was explicitly de-scoped without being run, not reported as an experimental negative. De-scoped without measured failure: #1405 stand-alone autoresearch loop, #1428 incompatible K24–64 sweep, #1324 hosted model intake, and #582 separate growth campaign. The parent adoption, safety, source-native E2 and release tasks remain open.

Canonical work queues: #200 session tracker, #1560 product graph, #1339 multilingual, #1582 no-fit, #1477 integrated task evaluation, #1346 external evidence, #1345 release sequence. A new experiment needs a precommitted hypothesis and source identity under one of these parents; avoid a new issue for every intermediate rerun.

7. Resume checklist

1. Read this page, the OSS landscape, original prior-art roadmap, product north star, and immutable experiment index.
2. Inspect current default branch, open PRs, terminal Actions artifacts, exact source SHA and #200. Never infer an outcome from CI green alone.
3. Fix P0 security/provenance if any new counterexample exists. Otherwise run one preregistered, bounded, single-variable P1 source-valid experiment on Linux CPU.
4. Report support recall and irrelevant false admission plus original actual agent utility. Negative outcomes are part of the paper; do not rerun until positive.
5. Keep v0.18 PR #1356 on HOLD until #1345/#1346 release-critical review. No new independent E2 label without actual source-disjoint authored data.

Existing research continuity: design history, prior-art evidence, paper evidence, and external validation freeze.