Accuracy answers the wrong question
Operational stakeholders do not ask whether the model was right. They ask whether the work got done, how often a human had to intervene, how long it took and what it cost.
The evaluation surface we build
A production evaluation harness usually covers five dimensions.
- Task completion against a business-defined outcome.
- Escalation quality — did it stop when it should have stopped?
- Latency distribution, not averages.
- Behavioural drift across model and prompt versions.
- Cost per completed unit of work.
References
- Argbit Eval design notes — Argbit Labs