Skip to main content

Spatial agents \u00b7 Findings

Vision models invent time limits on road signs that have none

We asked Gemini models whether each school-zone speed sign in street images was time-limited. Gemini 2.5 Flash said yes on 150 signs that were not, and 147 of those answers were the same text.

Research

In short

  • We asked Gemini models to read school-zone speed signs in street-level images from Berlin and Potsdam, and to say whether each one applies only at certain times.
  • Gemini 2.5 Flash said "time-limited" on 150 signs that the reference label marks as permanent. 147 of those 150 answers gave the same hours, "Mo-Fr 7-17h". The other 3 said "Mo-Fr 6-18h".
  • A step-by-step prompt cut the false alarms from 150 to 61, and for Gemini 3.1 Flash Lite from 135 to 25. It also cut recall: Gemini 2.5 Flash found 57% of the real time-limited signs instead of 94%.

Permanent signs the model called time-limited (share of 186, lower is better)

  1. Gemini 2.5 Flashoriginal prompt80.6
  2. Gemini 2.5 Flashstep-by-step prompt32.8
  3. Gemini 3.1 Flash Lite (preview)original prompt72.6
  4. Gemini 3.1 Flash Lite (preview)step-by-step prompt13.4
150, 61, 135 and 25 of the 186 signs the reference label marks as permanent. April 2026. Source: How we measured.

Why it matters

A school-zone sign in Germany often has a small plate under it that limits the 30 km/h to school hours. If a map says the limit applies all day when it applies only on weekday mornings, drivers and routing get it wrong. If it says the reverse, children are less safe. The plate is small, often 35 by 20 pixels in a dashcam frame.

The model did not say "I cannot read the plate". It wrote the hours a school-zone plate usually has. 147 identical answers is not reading. It is the model's prior about school zones, written into a field that looked like a reading.

How we measured

We took about 311 school-zone speed signs in Potsdam and south-west Berlin, each with street-level images and a reference label from the project's sample data saying whether the sign is permanent (186) or time-limited (129). We asked each model one question per sign with all its images: is this limit time-limited, and if so, when? The runs were in April 2026, on Vertex AI, with gemini-2.5-flash and gemini-3.1-flash-lite-preview.

The reference label is itself derived data, not a person looking at every sign, so read the counts as a strong pattern rather than exact error rates.

Model and promptFalse positivesPrecisionRecall
Gemini 2.5 Flash, original15040.5%93.6%
Gemini 2.5 Flash, step by step6153.1%56.6%
Gemini 3.1 Flash Lite, original13544.7%85.2%
Gemini 3.1 Flash Lite, step by step2560.9%30.5%

The same thing on one image

In a later run we gave five vision models one street image with a warning triangle and a 30 sign, and no plate under it. We asked for the speed, the sign type and the plate text, in a fixed JSON format. All five read "30" and the children sign correctly. Llama 4 Scout answered that there was no plate. Gemma 4 26B said it was unreadable. Mistral 3.1 24B, Kimi K2.6 and Kimi K2.7 Code each wrote a school-hours plate that is not in the image. This is one image, so it is an example, not a rate.

Claude Haiku 4.5 did the same on the back of a sign: shown two crops of a sign seen from behind, it named the sign both times and marked the reading as legible.

What worked

Give the model a way to say no. A format that only has room for a reading gets a reading. When we added answers such as "not readable" and "back of sign", each with a required reason, the invented readings stopped in our checks.

Ask two models and act only when they agree. Gemini 2.5 Flash and Gemini 3.1 Flash Lite, both with the step-by-step prompt, agreed on fewer signs, but when they agreed they were right more often: 65.4% precision, against 53.1% and 60.9% alone. The 99 signs where they disagreed go to a person.

Do not trust a confident field. The invented plates came back with the same confidence as the real ones. The check has to come from outside the model: a second reader, the sign geometry, or a person.

What this means if you use vision models on map data

If a model fills a field you need, it will fill it whether or not the image supports it. Measure false positives on images where the answer is "nothing there", not only on images where it is. If you are checking map attributes against imagery and want to know which model to trust, tell us about the work.