WG3 — SCE Working Group on AI Assistance, Reproducibility & Standards

Part of the Research Program for AI as Babel Fish for Structural Economics.

Chair: Alan Lujan (designated June 2026).

Scope¶

This working group addresses assisted use, institutional uptake, and sustainability: AI tools for model authoring and interpretation, grounded in the formal semantic core; standards for reproducibility and robustness; and governance of the language as a community standard.

Aims¶

Develop symbolically grounded AI tooling — in which a language model supplies informal reasoning but must satisfy the typed Bellman-core and formal knowledge base of WG1 — for verified translation from informal model descriptions to formal modular DP and valid language specifications. In parallel, develop the reproducibility infrastructure (automated robustness testing, referee/data-editor workflows, REMARK-style archives) and the governance arrangements required for the language to function as an accepted standard in the profession.

Scientific rationale¶

AI-assisted modeling is scientifically useful only if outputs are constrained by formal semantics rather than by prose pattern-matching. The symbolically grounded approach now emerging in AI-for-science (see prior art §5) treats a symbolic semantic layer as a hard constraint on neural components. Separately, a specification language achieves little without adoption: reproducibility standards, systematic robustness analysis, and transparent governance are the institutional mechanisms through which technical capability translates into professional practice. Together, these address the question of whether a published result depends on undocumented computational choices.

Current state¶

AI assistance (from `matsya`)¶

Matsya: a research RAG system for modular DP. It (a) answers DP-theory questions with grounded citations and (b) performs one-to-one translations between dolo-plus YAML and formal modular DP. The design principle is “correctness before fluency” — preferring verification against the formal substrate over plausible but unverified text.
A federation capability bridging DP theory across corpora (e.g., BufferStockTheory ↔ Kamihigashi–Stachurski) with “translate, don’t merge” discipline. Hosted API (matsya.econ-ark.org); session logging to study how researchers develop models.

Reproducibility and standards¶

An automated robustness testing framework — systematically checking whether economic conclusions depend on (seemingly benign) numeric choices — together with referee and policy-economist use cases.
REMARK already exists as an Econ-ARK standard (executable archive, pinned environment, reproduce.sh, CITATION.cff). Planned work includes a language-governance structure (PEP-13 as a model).

Cross-subfield translation and reproducibility¶

Two of WG3’s capabilities extend naturally across economics subfields (CGE, ABM, applied IO):

AI-assisted translation from subfield literatures. Matsya’s paper/code → formal-MDP → language pipeline can be pointed at the conventions of CGE, ABM, and applied IO — translating, for instance, a CGE equilibrium or an ABM rule set into (or out of) the shared representation — the bridge role for traditions that do not share the Bellman-core.
Cross-subfield reproducibility. Extending REMARK-style archives and automated robustness testing to results that span subfields, so a joined or cross-tradition model is reproducible and its sensitivity to computational choices is reported the same way as a single-tradition one.

The representational and numerical halves of cross-subfield integration sit in WG1 and WG2. See prior art §6 for cross-field precedents (OpenFermion as a translation bridge; the standing consortia behind OPTIMADE and ESMF).

Five-year milestones¶

Phase 1: Public, stable Matsya grounded in the WG1 typed core and formal knowledge base (citations to the IR and encoded definitions, not only to prose); evaluation of AI translation on the EconDP benchmark (joint with WG1); an automated robustness-testing prototype driven by WG2 methodization; seating of the Working-Group Advisory Board.
Phase 2: A verified paper → formal MDP → language spec pipeline (every AI-produced specification type-checked, round-tripped, and checked against the formal knowledge base where applicable); a reference library of REMARK reproductions expressed in the language; a pilot referee/data-editor workflow with at least one journal; published governance and standard decision records.
Phase 3: AI-assisted authoring as a routine part of the research workflow; the robustness/refereeing workflow integrated at one or more journals; a sustainable funding and governance model; recognition of the language as a standard for documenting structural-model methods.

Open research questions¶

Verified translation — LLM output grounded in typed semantics and the formal knowledge base, so that generated specifications are checkable rather than merely plausible.
Autoformalization — reliable model-spec → Lean translation, evaluated on the EconDP benchmark (joint with WG1).
Cross-paper DP-theory bridging; retrieval balance in federation (floor-weighting); workflow observation as research (what session logs reveal about how economists model).
Automated robustness testing — which numerical choices to vary, how to summarize sensitivity, how to present results to non-specialist referees.
Referee/data-editor workflow — design and evaluation of the journal-side process.
Governance design — adapting language-governance models (PEP-style) to a working-group-seated advisory board for economics.
Cross-subfield translation — adapting verified translation and reproducibility to the conventions of CGE, ABM, and applied IO (cross-cutting with WG1/WG2).
Sustainability — central-bank sponsorship, FRO conversion, subscription support.

Illustrative project proposals¶

Develop verified translation: a paper→formal-MDP→language pipeline in which every AI-produced specification is checked against the Bellman-core type system and the formal knowledge base, evaluated on the EconDP benchmark of published structural models.

Pilot the referee workflow: in partnership with a journal data editor, express a set of recently published structural results in the language, run automated robustness tests over their declared numerical methods, and report which conclusions are sensitive to computational choices.

Indicators of progress¶

Active Matsya users and sessions; translation accuracy (parses, round-trips, verification against the formal knowledge base); EconDP benchmark scores; REMARK reproductions in the language; journals and data editors piloting the workflow; robustness reports produced; governance documents ratified; sustained funding secured.

Potential funding sources¶

JHU DS/AI Institute, NSF (CISE / robust AI for science; open science / replication / cyberinfrastructure), central banks (robust policy models), FRO/Convergent, philanthropic follow-on.

Foundations¶

Sargent–Stachurski (the DP-theory corpus Matsya grounds in); the Bellman calculus and formal knowledge base (WG1) as the symbolic substrate; symbolically grounded AI (the broader paradigm, a.k.a. neuro-symbolic); BufferStockTheory (canonical examples); the Econ-ARK REMARK standard; Backus FFP (the meaning-vs-representation split that makes robustness testing well-defined); the econ-ark.org reproducibility mission.