Benchmarks

ESE-1 · evidence contract

Spec only · no certified results

Execution trace anatomy

A trace must show enough state to determine what the agent knew, what it attempted, what changed, why it retried, and whether the final claim is supported by authoritative systems.

Why traces matter

An answer log cannot explain a system change

Chat transcripts omit the facts that determine whether an enterprise action was correct: source versions, tenant identity, policy state, authority, idempotency scope, side-effect receipts, and authoritative read-back. Tool-call logs alone are also insufficient because a valid request can still produce the wrong business state.

The trace is the evidence boundary between an agent claim and an independently verifiable run. It should support replay of the scorer and reconstruction of every consequential transition without exposing customer secrets.

Required event sequence

From job envelope to terminal evidence

  1. 01

    Job envelope

    Job ID, tenant boundary, goal, attempt, model version, execution-graph version, and policy version.

  2. 02

    Checkpoint bind

    Source handles, versions, freshness, selected state, excluded state, and the reason each field is required.

  3. 03

    Decision

    Selected transition, permitted alternatives, active constraints, and refusal or escalation conditions.

  4. 04

    Tool call

    Tool version, typed arguments, idempotency key, authority, expected side effect, and timeout boundary.

  5. 05

    Observed result

    Raw status, normalized output, side-effect receipt, changed record versions, and classified error.

  6. 06

    Postcondition

    Authoritative read-back, invariant checks, next legal transitions, and state committed to the run.

  7. 07

    Recovery

    Failure classification, preserved progress, compensation or retry decision, and escalation payload.

  8. 08

    Terminal evidence

    Final system identifiers, state digest, satisfied checks, unresolved exceptions, and scorer decision.

Schema v0.1 records a strict total order within each run attempt. Parallel calls receive distinct sequence numbers at dispatch and preserve parent checkpoint identifiers for causal reconstruction.

Synthetic specimen

The timeout that must not become a duplicate payment

This schema specimen illustrates the evidence shape; it is not presented as a production result. The bank timed out after accepting the request, so the only safe transition is verification—not blind retry.

{
  "event": "tool_call",
  "run_id": "TRS-PAY-02/attempt-01",
  "checkpoint": "payment.release",
  "bound_state": {
    "beneficiary_version": "ben_17:v4",
    "policy_version": "pay_policy:v12",
    "approval_state": ["approver_a", "approver_b"]
  },
  "tool": "banking.payment.release@2",
  "idempotency_key": "sha256:…",
  "expected_postcondition": "payment.status == RELEASED",
  "result": "TIMEOUT_AFTER_ACCEPT",
  "next_transition": "VERIFY_BEFORE_RETRY"
}

A retry is legal only after an authoritative lookup proves that no payment exists for the idempotency scope. A successful second API call is not evidence that the overall job was correct.

Redaction contract

Remove secrets without removing causality

Enterprise traces can be useful without exposing customer values. Redaction fails when it makes the run impossible to audit.

  • Redact business values, personal data, credentials, and customer-specific policy text.
  • Preserve record relationships, event order, versions, tool identity, error class, and state-transition semantics.
  • Hash stable identifiers consistently within a run so cross-system reconciliation remains inspectable.
  • Publish every failed attempt that contributed to the terminal state; do not present only the successful retry.

Publication checklist

What a reviewer should be able to answer

Which state was authoritative at each checkpoint?
Which tool versions and permissions were active?
Did any call produce an uncertain or partial side effect?
How did the runtime decide whether a retry was safe?
Which invariants were checked after each write?
What exact evidence caused the scorer to pass or fail?