Download

* Equal contribution.


Abstract

LLM agents now act on real systems, and running the same task more than once is increasingly routine. Traditional metrics such as pass rate score how often the agent succeeds but say nothing about whether those runs agreed on how they got there. We propose a complementary axis, behavioral consistency: a property of the trajectory distribution from which an agent draws when invoked on the same task repeatedly. We formalize it as a totally ordered hierarchy of five levels and a metric ConsistencyNlv for the probability that N independent runs agree at level lv. Across 17 models on 10 system-administration tasks, pass rates routinely exceed 80%, yet two random runs agree on the sequence of state-modifying actions (i.e., Consistency2lv3) only about 39% of the time on average. Even the frontier model reaches only Consistency2lv3 ≈ 57%. Case studies on certificate rotation and zero-downtime upgrades further show that the same pass rate can hide trajectory distributions with very different deployment-risk profiles.