Computer vision
How to Benchmark Image Embeddings for Search
Compare image embeddings using relevant-item labels, recall at k, ranking quality, latency, and dataset slices.
A high similarity score does not guarantee useful image search. Evaluate whether the right images appear near the top for the searches your users make.
Build a labeled retrieval set
Choose queries such as a text description, a reference image, or a product attribute, then mark which catalog images are relevant. Include near duplicates, visually similar but wrong items, and rare categories. Split by product or source where duplicates could leak between development and test sets. If text and image queries are both supported, score them separately.
Measure rank quality
Report recall at k: for how many queries does a relevant result appear in the first k? Add a rank-sensitive measure such as nDCG when more than one relevance level exists. Show per-category results and failure examples. The Sentence Transformers retrieval evaluator documentation describes query, corpus, and relevance-label inputs that form a useful evaluation pattern, even when the embeddings are for images.
Include the search system
Keep the vector index, distance metric, preprocessing, and reranking policy fixed when comparing encoders. Record embedding generation time, index size, query latency, and update cost. A model with better offline ranking may still be unsuitable if it cannot meet the intended catalog refresh or response-time budget.