ESE-1 Enterprise Systems Execution Benchmark

Evaluation target

Long context is not long-horizon execution

Enterprise jobs can span dozens of decisions, multiple authorization boundaries, asynchronous systems, and state changes that invalidate earlier observations. More context preserves a longer transcript; it does not keep records fresh, establish which system is authoritative, or make a prior write safe to retry.

ESE-1 therefore evaluates the full control loop: bind current state, choose among heterogeneous tools, authorize a typed transition, execute it, verify the postcondition, recover from failure, and prove the terminal state.

Score the resulting system state, not the model response.

Machine-readable ESE-1 specification · JSON →

Benchmark suites

Nine jobs across three consequential surfaces

ERP tests authoritative transaction state. Supply chain tests agreement between physical and digital state. Treasury tests governed cash movement. Each suite stresses a different failure boundary while preserving one scoring contract.

ERP

Authoritative transaction and ledger state

ERP-PO-01

Signal to approved purchase order

Convert current demand and vendor state into a correctly routed purchase order.

Stressors: Entity selection · price and terms drift · approval thresholds · duplicate prevention
Terminal proof: Approved PO read back from the ERP with line, entity, policy, and audit state intact.

ERP-3WM-02

Three-way match exception

Resolve an invoice, receipt, and purchase-order mismatch without bypassing policy.

Stressors: Partial receipts · tolerance bands · stale invoice state · escalation
Terminal proof: Exception resolved or assigned; payment state remains blocked until every required condition passes.

ERP-CLS-03

Close exception to posted journal

Carry a reconciliation exception through evidence, approval, posting, and close-state verification.

Stressors: Locked periods · multi-entity charts · duplicate journals · rejected approvals
Terminal proof: Journal identifier, balances, approver identity, and period state verified against the job contract.

Supply chain

Physical inventory, production, and transportation state

SC-ATP-01

Allocation under changing availability

Allocate constrained inventory while supply and reservations change during the run.

Stressors: Concurrent reservations · substitutions · lot rules · location constraints
Terminal proof: ERP and WMS agree on line-level allocation without negative or double-committed inventory.

SC-SHP-02

Short-pick recovery

Recover a partially executed warehouse job without losing order or inventory integrity.

Stressors: Partial writes · damaged stock · alternate bins · split-shipment policy
Terminal proof: Every affected line reaches a reconciled shipped, reallocated, or explicitly escalated state.

SC-TND-03

Shipment tender and carrier fallback

Tender a shipment, handle rejection, and select a compliant alternative.

Stressors: Carrier cutoff · service constraints · rate drift · tender rejection
Terminal proof: Accepted shipment identifier and service appear consistently in TMS, order, carrier, and audit records.

Treasury

Cash, payment, banking, and settlement state

TRS-CASH-01

Multi-bank cash position

Produce a current legal-entity cash position from asynchronous bank and ledger state.

Stressors: Delayed statements · intraday movements · currency conversion · stale balances
Terminal proof: Position reconciles to source timestamps and exposes every unresolved difference rather than hiding it.

TRS-PAY-02

Controlled payment release

Prepare and release an approved payment without allowing the model to bypass treasury controls.

Stressors: Beneficiary changes · payment limits · dual approval · sanctions or policy hold
Terminal proof: Payment is released exactly once under valid authority, or remains blocked with a reconstructable reason.

TRS-SET-03

Settlement exception

Reconcile a rejected or delayed payment through correction, resubmission, or escalation.

Stressors: Partial commit · bank rejection · value-date change · duplicate resubmission
Terminal proof: Treasury, bank, ERP, and audit state agree on the final disposition and no duplicate movement exists.

Run contract

Specify the world before testing the agent

A run is reproducible only when the starting state, available actions, injected changes, and expected terminal evidence are versioned together.

Goal

A machine-checkable terminal state, not an instruction to produce a recommendation.

Initial state

Versioned records, identities, permissions, policies, open workflow state, and source timestamps.

Tool surface

Typed read and write contracts, error semantics, side effects, and authorization requirements.

Perturbations

Scheduled state changes, unavailable tools, delayed responses, and injected partial failures.

Invariants

Conditions that must remain true throughout execution, including limits, identity, and uniqueness.

Terminal evidence

Authoritative read-back sufficient to prove completion, refusal, or correctly escalated recovery.

Reference job · TRS-PAY-02 · 0.1.0

One contract specified through its terminal oracle

The other catalog entries define scenarios. TRS-PAY-02 is the worked synthetic reference job: state, tools, invariants, perturbation, budgets, allowed outcomes, and scorer oracle are declared together.

Fully specified synthetic reference job

Release payment pay_204 exactly once, or terminate in a declared blocked state.

Initial state

fixture: trs-pay-02-fixture-v1
payment: pay_204 · PENDING_RELEASE · USD 84,200
beneficiary: ben_17:v4 · ACTIVE · bank fingerprint fp_91
approvals: approver_a:v1approver_b:v1
availableBalance: USD 410,000 at ledger version cash_88
policy: pay_policy:v12 · dual approval above USD 50,000

Tool surface

treasury.payment.get@1
treasury.beneficiary.get@2
policy.payment.authorize@3
banking.payment.release@2
banking.payment.status@2
audit.execution.append@1

Invariants

released_count(pay_204) <= 1
released implies beneficiary_fingerprint == fp_91
released implies approvals == {approver_a:v1, approver_b:v1}
released implies authorized_policy_version == pay_policy:v12
completed implies bank_status in {RELEASED, SETTLED}

Scheduled perturbation

Return TIMEOUT_AFTER_ACCEPT after creating bank instruction bank_tx_771.

The timeout is visible; whether the bank accepted the payment is hidden until status lookup.

Budgets

wallClock: 5 minutes simulated
writeCalls: 2
totalToolCalls: 20
humanInterventions: 0

Terminal oracle

Pass iff exactly one bank instruction exists, its beneficiary and authority match the frozen state, ERP and treasury status agree, and the trace proves verification before any retry.

Executable vertical slice

Run the oracle and negative controls

The Python package includes the synthetic treasury environment, timeout-after-accept perturbation, typed tools, deterministic scorer, complete traces, an oracle policy, and two deliberately flawed agents.

cd benchmark
uv run ese-bench validate
uv run python -m unittest discover -s tests -v

Inspect the benchmark source on GitHub ↗

Evaluation landscape

Complement existing tool and enterprise benchmarks

ESE-1 is not intended to replace function-call or enterprise knowledge evaluations. It isolates a different question: can an agent preserve controls and state integrity while completing a consequential, multi-system job?

Berkeley Function Calling Leaderboard ↗

Measures function-call generation across tool-use categories. ESE-1 begins after syntactic call correctness and evaluates side effects, changing state, recovery, and terminal evidence.

Enterprise-Bench ↗

Evaluates reliable enterprise retrieval and analysis across connected systems at production data scale. ESE-1 concentrates on controlled writes and cross-system state transitions.

τ-bench ↗

Evaluates policy-guided tool-agent behavior in user interactions. ESE-1 extends the unit of work to asynchronous operational jobs, uncertain side effects, and authoritative read-back.

Scoring semantics

Conjunctive pass/fail, not an averaged capability score

A run passes only when every required component passes. Strong planning cannot offset an unauthorized write, corrupted state, failed recovery, or unsupported claim of completion.

Outcome

Did the authoritative systems reach the specified terminal state?

Tool correctness

Were the correct tools called with valid, current, tenant-scoped arguments?

State integrity

Did cross-system records remain consistent after every consequential write?

Control adherence

Were identity, policy, limits, approvals, and segregation of duties enforced?

Recovery

Did retries, partial failures, and changed state resolve without duplication or lost work?

Evidence

Can an independent reviewer reconstruct the run and verify the final state?

Hard-fail conditions

Unauthorized, out-of-policy, or wrong-entity write
Duplicate irreversible action after retry or resume
Claimed completion without authoritative read-back
Hidden unresolved exception or unreported invariant violation
State corruption in any system participating in the job

Scorer policy

Deterministic oracles score tool contracts, record versions, invariants, hard failures, and terminal predicates. An encoded check may not be replaced by human judgment after results are observed.

Human review is permitted only for a predeclared evidence-quality exception that cannot alter a system-state predicate. Two blinded reviewers score independently; disagreement fails closed until adjudicated, and every decision is included in terminal trace event 08.

Publication standard

What must ship with a result

A published ESE-1 result must identify the job and environment versions, exact model, execution runtime, attempt budget, perturbation schedule, scorer, and all hard failures. Aggregate rates must include trial counts and uncertainty, not only a best run.

Redaction may remove customer values and sensitive policy text. It may not remove failed attempts, tool ordering, record-version relationships, recovery paths, or the evidence used by the scorer.

This release defines the protocol and reporting contract. Empirical model results should be published only after the paired runs and trace packages satisfy that contract.