Why It Matters
Most benchmarks hide this variability behind an average.
Overview
That is embarrassing onstage. In production, it is a reliability problem: a workflow that succeeded once may fail the next time a user makes the same request. For mission-critical work, such as reconciling a financial transaction or checking a contract for an obligation, that can be a showstopper.
Why It Matters
Most benchmarks hide this variability behind an average.
Key Details
Most benchmarks hide this variability behind an average.
On AppWorld, a ReAct agent using GPT-4.1 succeeded on 77.4% of runs across five repetitions.
But it succeeded in all five runs for only 53.0% of tasks — a 24.4-point consistency gap.
Most benchmarks report the first number.
We built a way to measure the second — and improve it.
--- *Original report published by [Hugging Face Blog](https://huggingface.co/blog/ibm-research/altk-evolve-consistency).*
