Innnodata finds leading AI models limited in visual detail recognition
Innnodata evaluated seven AI models on 150 visual‑thinking tasks and found that the top model Gemini 3.1 Pro chose a distractor in 85.7% of its errors on rotation and analogy problems
Key points
- The company tested seven models – five leading API models and two open‑weight models – on 150 items, 25 per category.
- The assessment revealed two separate requirements for visual understanding: the ability to infer image structure and the ability to perceive fine details accurately.
- If a client prioritises precise recognition of tiny visual elements, Innodata’s results can serve as a benchmark for choosing or refining models, reducing error risk in critical applications
What happened
The company tested seven models – five leading API models and two open‑weight models – on 150 items, 25 per category. Three leading models (Gemini, Claude, GPT‑4o) scored 0.70–0.79 overall, far above the 20% chance level. The highest sub‑category scores in analogy and rotation were 0.56. Gemini 3.1 Pro identified the pattern but chose the distractor in 85.7% of cases, missing that the only difference was the flag’s orientation
Why it matters
The assessment revealed two separate requirements for visual understanding: the ability to infer image structure and the ability to perceive fine details accurately. Gemini 3.1 Pro showed weakness in the second requirement, failing to distinguish a slight change in flag orientation, which led to a high distractor‑selection rate. Clients using visual analytics for medical diagnosis or autonomous driving could suffer errors if their model has similar flaws
When it can help
If a client prioritises precise recognition of tiny visual elements, Innodata’s results can serve as a benchmark for choosing or refining models, reducing error risk in critical applications
Risks and uncertainties
A high distractor‑selection rate in leading models could prompt the market to doubt Innodata’s evaluation credibility, potentially harming its reputation as an AI‑benchmark provider
What to watch
Future disclosures, such as in a Form 10‑Q, may contain data on the commercial rollout of the Agent Personalization Platform and whether customers have begun using it actively
Original sources
- Beyond the Caption: What Building Our Own Visual-Logic Benchmark Taught Us About Today’s VLMs
- Why Personalization Is a Training Problem for Long-Horizon AI Agents
Cite this article
The fingerprint identifies the exact version of each original that the article was written from; the original stays at its source.
Follow Innodata
A notification in your browser or on your phone when we publish about Innodata. Free, no email, no account, no app.
Prepared by AI from official company documents; automatic checks and an AI editor review it before publication, and not every article is read by a person. This is not investment advice. Editorial standards
