From Model Uncertainty to Human Attention

Abstract

AI-assisted annotation has turned large-scale labeling into a review task. Instead of annotating from scratch, humans inspect model-generated predictions and decide, object by object, where to direct their bounded attention. Lacking any signal of where a prediction is actually unreliable, annotators tend to focus on objects that stand out visually. This creates a challenge for object detection, where slightly misaligned bounding boxes can appear almost identical to accurate ones, even when the model is highly uncertain about the object's true boundaries. Furthermore, a detector can be confident about an object's class while being uncertain about its boundaries. Existing work uses model uncertainty to decide which images reach an annotator and to signal confidence in an object's class. However, it does not address how attention should be allocated within an already labeled image to ensure accurate object localization.

We surface a detector's localization uncertainty in an AI-assisted annotation interface as a color-coded visual cue and explore its effects on annotators. In a randomized controlled study, 120 participants completed 1,800 annotation trials on autonomous-driving images. A box-level analysis shows that the cue successfully reallocates effort. Participants focus their edits on the model's least reliable boxes, reversing the pattern observed in the unguided baseline, where annotators disproportionately adjust boxes that are already well-placed. This reallocation improves both outcomes at once, raising label quality by 0.70 points of mIoU while making annotators 7.2% faster, with improvements concentrated on harder images and no measurable increase in cognitive load. Localization uncertainty is thus an effective interaction design lever wherever bounded human attention must be allocated across model outputs whose errors carry no perceptual signal. Code is available at https://mos-ks.github.io/MUHA/.

Interactive viewer

Step through the 97 KITTI images used in the experiment and compare the three annotation layers reported in the paper. Toggle layers on or off, jump between difficulty bins, or use the ← / → keys to navigate.

Loading…

What am I looking at?

Original KITTI labels are the boxes that ship with the dataset. They are sometimes imprecise, missing for occluded/distant objects, or mis-labelled.
Detector predictions come from the probabilistic EfficientDet-D0 model. Toggle “Colour by uncertainty” to map the per-coordinate aleatoric localization uncertainty onto the box border (blue = certain, red = uncertain).
Re-annotated ground truth is the gold standard the three authors produced after independently re-labelling thirds of the experimental pool. It corrects 41 erroneous labels and adds 430 missed annotations.

Paper

A pre-print PDF will be linked here once available. In the meantime please use the BibTeX entry below.

@unpublished{KaSbHoSpNaSa2026,
  title  = {From Model Uncertainty to Human Attention:
            Localization-Aware Visual Cues for Scalable Annotation Review},
  author = {Kassem Sbeyti, Moussa and Holstein, Joshua and Spitzer, Philipp
            and Klein, Nadja and Satzger, Gerhard},
  note   = {Manuscript under review},
  year   = {2026}
}

Data & code

Labeling tool — a Streamlit app that participants used in the study, with a relabel_* mode used by the three authors to produce the re-annotated ground truth. View on GitHub →
Detector predictions with per-coordinate aleatoric localization uncertainty (data/detector_predictions.txt).
Re-annotated ground truth covering the experimental image pool (data/relabeled_ground_truth/).
Difficulty bins stratifying the candidate pool by mean per-image localization uncertainty (data/{low,mid,high}_uncertainty.txt).

The KITTI subset shipped with this repository is redistributed under the CC BY-NC-SA 3.0 license. For commercial use, please obtain the data directly from the KITTI website. See LICENSE-DATA.md in the repository for the full attribution.