Computer vision
How to Evaluate Image Captioning Models Beyond a Single Score
A practical image captioning evaluation plan covering factual accuracy, missing details, accessibility, and human review.
A caption can sound fluent while describing an object that is not in the image. An evaluation therefore needs to check grounded accuracy and usefulness, not just similarity to a reference sentence.
Choose images for the intended use
Build a held-out set with the image types your application will see: product photos, screenshots, diagrams, crowded scenes, low-light images, and images containing text. Include cases where a safe caption should say that a detail is unclear. Keep any private images permissioned and separate from model tuning.
Score what the caption says
Have reviewers mark each caption for correct objects, actions, relationships, and important omissions. Count invented details separately from missing ones; an invented medication label or road sign can be more harmful than an incomplete description. For accessibility, ask whether the caption helps a reader understand the image in context and avoids speculation about people's identity or intent.
Automatic overlap scores can be useful for repeatable screening, but many valid captions use different words. The Transformers image captioning guide demonstrates a reference-based evaluation workflow. Pair that with blind human review of a sample, especially disagreements and hallucinations.
Make the comparison reproducible
Use the same images, prompts, output length limits, and decoding settings for each candidate. Report results by image type, plus caption latency and failure examples. A single aggregate score may conceal a model that performs well on photographs and poorly on screenshots.