Hypothesis
Model independence is an empirical claim
An abstraction layer around provider APIs does not make models interchangeable. The relevant question is whether a model change preserves end-to-end completion when the job requires long-horizon state management, diverse tool selection, policy compliance, and recovery from partial writes.
The test must isolate inference from the rest of the system. If prompts, workflow branches, tool definitions, or scoring rules are quietly rewritten for each provider, the result measures separate implementations—not a model swap.
Held constant
- =Job specification, initial state, permissions, and injected perturbations
- =Tool schemas, adapter behavior, policy engine, and transaction boundaries
- =Context-bind rules, recovery machinery, terminal checks, and scorer
- =Inference profile: temperature, top-p, reasoning mode, parallel-call policy, and output-token limit
- =Attempt budget, timeout policy, human-intervention policy, and reporting format
Allowed to vary
- ΔFoundation-model provider and exact model version
- ΔProvider-native tool-call representation and adapter translation
- ΔInference latency, token use, and model-returned reasoning artifacts when available
Paired-run method
One release, matched conditions, every outcome
- 01
Freeze the execution release
Version the job, state fixture, tool adapters, policies, bind logic, recovery rules, and scorer as one immutable release.
- 02
Generate a balanced trial schedule
Run every model against the same state variants and perturbations; randomize order so time-dependent services do not favor one provider.
- 03
Execute within identical budgets
Apply the same attempt, tool-call, timeout, and human-intervention limits. A provider-specific exception must be disclosed.
- 04
Score authoritative outcomes
Use deterministic system-state checks for every terminal predicate; blinded review is limited to predeclared evidence-quality exceptions.
- 05
Publish every paired result
Report passes, hard failures, refusals, escalations, and incomplete runs. Do not discard trials after an inconvenient model response.
Reporting matrix
Completion, safety, recovery, and cost belong together
A lower token bill is not an improvement if the model requires more retries, escalates routine cases, or reports completion before the system reaches the terminal state.
End-to-end job pass rate with confidence intervals
Critical write violations and unsafe false-completion rate
Tool selection, argument, and sequencing errors
Recovery success after injected partial failure
Median and tail tool calls, elapsed time, input tokens, and output tokens
Human interventions and unresolved escalations per completed job
Analysis plan
Analyze paired outcomes, not disconnected percentages
Predeclare the margin
Set the non-inferiority margin, minimum trials per job, stopping rule, and unacceptable safety regression before observing results.
Preserve the pairing
Compare models on matched initial states and perturbations. Report discordant pairs and a paired confidence interval, not only aggregate pass rates.
Stratify by job
Report every job and stressor separately before any macro average so easy, frequent cases cannot hide a failed consequential workflow.
Repeat stochastic trials
Use multiple declared seeds or repeated provider calls per fixture and report variance rather than selecting the cleanest trajectory.
Pending before the first result release: freeze the absolute non-inferiority margin, α, minimum paired trials per model × job, interval method, sequential stopping rule, and multiplicity policy. No model-swap claim is valid while those fields remain unset.
Claim discipline
What the result may—and may not—support
“Execution survives the model swap”
The replacement model remains within the predeclared non-inferiority margin for job completion and introduces no unacceptable increase in critical writes.
“The model is interchangeable”
The same runtime, job contract, controls, and scorer operate without model-specific workflow code or hidden prompt retuning.
“The model is more efficient”
Cost or latency improves without shifting failures into human intervention, refusal, or unsafe false completion.
Until measured runs, trial counts, confidence intervals, and traces are published, this page defines the evaluation contract rather than asserting a winner.