Comfortably Stuck

Overview

Closed-loop evaluation is widely regarded as a more faithful way to assess end-to-end driving models than open-loop evaluation. We find that Bench2Drive's evaluation protocol can itself misrepresent the models scored on it.

The benchmark's comfort metric mostly measures how long a vehicle stays stationary: fixed one-second segments keep accumulating while a vehicle is stopped. We propose a distance-based segmentation, which scores comfort over the distance actually driven. We also identify two navigation-signal pitfalls: the default controller can replace model-predicted steering with a route-target angle, and the evaluation target point can differ substantially from the training input.

When “comfort” rewards standing still

The original metric only rises while a car is stopped, even though it has made no progress. Distance-based segmentation scores the drive itself instead.

Side by side: fixed one-second segments in the original metric keep accumulating while the car is stationary; distance-based segments advance only as it drives.

r = 0.97

Traditional comfort tracks time spent stationary across eight models.

r = 0.06

Distance-based comfort removes that relationship.

2 pitfalls

Controller steering overruling and target-point train/evaluation mismatch.

Navigation-signal pitfalls in action

Steering overruling can mask a model's own control behavior, while target-point sampling can give a model inputs unlike those used in training. Their effects differ across models, so the default scores are not directly comparable without checking how the harness supplies navigation signals.

Steering overruling (SO). With SO enabled, the controller overcorrects and pushes the car off the road.

Target-point mismatch (TPM). Because the target point during evaluation is sampled differently than during training, the model sees a wrong future LANEFOLLOW command and misses the turn.

Both issues fixed. The model receives the corrected signals and successfully makes the turn.

Illustration of steering error compensation, a blocked overtake, and the difference between training-time and test-time navigation commands. — The controller may steer toward the route target rather than follow the model's waypoints; route-target sampling may also shift between training and evaluation.

Comfort results

Closed-loop results on the 220-route Bench2Drive benchmark. UniAD and VAD use the corrected navigation protocol described in the paper.

Method	Traditional comfort	Fixed comfort implementation	Distance-based comfort	Stationary time
UniAD	39.28	79.06	72.93	45.67%
VAD	45.95	56.84	37.21	47.64%
SparseDriveV2	7.75	23.50	50.28	7.08%
TFv6	19.58	25.97	29.20	19.83%
Orion	14.76	70.24	75.98	23.07%
SimLingo	31.71	36.80	25.86	32.16%
CaRL	22.81	61.36	71.84	24.76%
PDM-Lite	22.77	32.57	35.82	20.43%

Higher comfort scores indicate more segments passing the comfort thresholds. The traditional comfort metric in Bench2Drive scores driving comfortable exactly when the car is stationary. Fixing implementation errors results in driven segments being correctly classified as comfortable. Only distance-based segmentation is not inflated when the vehicle is stationary.

Citation

@misc{aghajanzadeh2026comfortably,
  title  = {Comfortably Stuck: Evaluation Pitfalls in Bench2Drive's Closed-Loop Protocol},
  author = {Aghajanzadeh, Emad and Schwarzfischer, Thomas and Thoma, Moritz and
            Fasfous, Nael and Vemparala, Manoj Rohit and Mori, Pierpaolo and
            Mueller-Gritschneder, Daniel and Schlichtmann, Ulf},
  year   = {2026},
  eprint = {2610.21741},
  archivePrefix = {arXiv}
}