Insider
Innodata IncINOD

Innnodata finds leading AI models limited in visual detail recognition

Innnodata evaluated seven AI models on 150 visual‑thinking tasks and found that the top model Gemini 3.1 Pro chose a distractor in 85.7% of its errors on rotation and analogy problems

Verified against 2 archived primary sources · SHA-256 recorded Inspect sources & citations ↓
MacBook Pro on table beside white iMac and Magic Mouse
Photo by Domenico Loia on Unsplash
85.7%
Error distractor rate
0.70–0.79
Overall score range
150 items
Test items

Key points

  • The company tested seven models – five leading API models and two open‑weight models – on 150 items, 25 per category.
  • The assessment revealed two separate requirements for visual understanding: the ability to infer image structure and the ability to perceive fine details accurately.
  • If a client prioritises precise recognition of tiny visual elements, Innodata’s results can serve as a benchmark for choosing or refining models, reducing error risk in critical applications

What happened

The company tested seven models – five leading API models and two open‑weight models – on 150 items, 25 per category. Three leading models (Gemini, Claude, GPT‑4o) scored 0.70–0.79 overall, far above the 20% chance level. The highest sub‑category scores in analogy and rotation were 0.56. Gemini 3.1 Pro identified the pattern but chose the distractor in 85.7% of cases, missing that the only difference was the flag’s orientation

Why it matters

The assessment revealed two separate requirements for visual understanding: the ability to infer image structure and the ability to perceive fine details accurately. Gemini 3.1 Pro showed weakness in the second requirement, failing to distinguish a slight change in flag orientation, which led to a high distractor‑selection rate. Clients using visual analytics for medical diagnosis or autonomous driving could suffer errors if their model has similar flaws

When it can help

If a client prioritises precise recognition of tiny visual elements, Innodata’s results can serve as a benchmark for choosing or refining models, reducing error risk in critical applications

Risks and uncertainties

A high distractor‑selection rate in leading models could prompt the market to doubt Innodata’s evaluation credibility, potentially harming its reputation as an AI‑benchmark provider

What to watch

Future disclosures, such as in a Form 10‑Q, may contain data on the commercial rollout of the Agent Personalization Platform and whether customers have begun using it actively

Original sources

  1. Beyond the Caption: What Building Our Own Visual-Logic Benchmark Taught Us About Today’s VLMsSeptember 17, 2026SHA-256 of our archived copy 2883c4c8f264…
  2. Why Personalization Is a Training Problem for Long-Horizon AI AgentsOctober 6, 2026SHA-256 of our archived copy b587decbdf57…
Cite this article

The fingerprint identifies the exact version of each original that the article was written from; the original stays at its source.

Prepared by AI from official company documents; automatic checks and an AI editor review it before publication, and not every article is read by a person. This is not investment advice. Editorial standards