An image may be worth a thousand words, but most captioning models describe it in only a few. Modern vision-language models produce fluent high-level captions, yet routinely miss the attributes, counts, textures, materials, and spatial relations that make an image visually specific. Recent multi-stage systems recover some of these details through generation, decomposition, verification, and rewriting — at the expense of substantially higher inference latency.
We propose SimLoss, a reference-free embedding-space objective for single-pass fine-grained image captioning. SimLoss trains a vision-language model to align its projected hidden-state representation with a frozen image embedding through an InfoNCE contrastive loss, supplying a dense visual supervision signal before any text is decoded — and requiring neither human-written fine-grained captions nor pseudo-captions from a multi-stage pipeline. We instantiate it as SimLoss FFT, which backpropagates through a locally available embedding model, and SimLoss GRPO, which treats that model as a black-box reward.
Compared with single-pass, multi-stage verification, reward-optimized, and perception-aware baselines, SimLoss FFT achieves the highest precision while nearly matching the F1 of the multi-stage method — all while retaining single-pass inference and running roughly 20× faster than the multi-stage pipeline. SimLoss GRPO attains the strongest recall. Together, these results show that embedding-space supervision can recover the quality of multi-stage verification at the latency of a single-pass captioner.
Humans can glance at an image once and retain fine visual detail — the material of a lamp base, the pattern on a fabric, the number of repeated objects. Vision-language models, despite their fluency, often miss this level of specificity: “a lamp on a table” is not wrong, just incomplete. The task is not to generate longer captions — asking for more detail is known to increase hallucination. It is to improve visual coverage while maintaining factual grounding.
Standard caption supervision is not designed for this setting. MS COCO captions are intentionally concise — 10.0 words on average in our sampled batch — while the hyper-detailed human descriptions of IIW-400 average 171.2 words, roughly 17× longer. Multi-stage pipelines can generate detailed pseudo-captions, but not clean supervision: verified CapMAS captions average 29.3 atomic propositions per image, of which 7.0 are judged false — a factuality ratio of 0.766. Distilling from them would transfer their unverifiable claims and stylistic artifacts along with their detail.
View a fine-grained captioner as preserving discriminative visual information. A generic caption remains compatible with many similar images; a detailed caption should make the source image easy to identify. Ideally a caption ŷ maximizes the mutual information
but this is intractable over discrete captions. SimLoss applies the same principle before decoding, in continuous representation space: align the VLM's pooled hidden state with a frozen image embedding that already captures visual similarity — no caption targets at all.
Embedding-space similarities in a frozen Qwen3-VL-Embedding space sharpen the motivation. The base captioner is not disconnected from the image — its generations already sit close to the source image (0.6982 cosine). The gap is between what supervision provides and what evaluation demands: COCO's short human captions align with their own images at only 0.4794, far below the fine-grained regime IIW evaluates. SimLoss closes this gap with the image embedding itself, rather than with fine-grained human captions or pipeline-generated captions as targets.
| IIW human | CapMAS | Qwen2.5-VL | |
|---|---|---|---|
| Image | 0.6616 | 0.7000 | 0.6982 |
| IIW human | — | 0.6928 | 0.6803 |
| CapMAS | — | 0.7307 |
| COCO human | Qwen2.5-VL | |
|---|---|---|
| Image | 0.4794 | 0.6982 |
| COCO human | — | 0.5055 |
During training, a frozen multimodal embedding model encodes the image while the trainable VLM processes the same image and prompt. The VLM's hidden states are mean-pooled, projected into the embedding space, and aligned contrastively with the frozen image embedding. At inference, the embedding model and projector are removed — captioning uses only the adapted VLM, in a single pass.
Let EI be the frozen image encoder and gφ a learned two-layer MLP projector from the VLM hidden dimension into the image-embedding space. For a batch of N image–prompt pairs with ziI = EI(vi), ziV = gφ(hθ(vi, x)) and cosine similarities sij = cos(ziI, zjV), SimLoss optimizes the InfoNCE loss
A generic representation that discards visual detail is hard to match uniquely to its source image among in-batch alternatives; a representation that preserves fine-grained attributes, counts, textures, materials, and spatial relations aligns strongly with the correct image embedding. SimLoss therefore rewards the captioner for retaining discriminative visual information before text generation occurs — no human-written fine-grained captions, no pseudo-captions, and, when the embedding model is differentiable, no sampling of captions during training.
local embedding modelgradient supervision
Gradients flow from the InfoNCE loss through the projector into the LoRA adapters; the base weights and encoder stay frozen. “Fully differentiable” describes the training signal, not full-parameter tuning. The signal is dense and applied before any discrete text is sampled.
best precision · 0.8485shortest captions · 114.9 ± 13.1 words
API-only embedding modelreinforcement learning
When the embedding model offers no gradients (here: Gemini 2 Embed), sampled captions are scored with the reward rSim(v, ŷ) = cos(zI, zT) — image and caption embedded in the same space — and the policy is optimized with GRPO over sampled caption groups.
best recall · 0.6015best CLAIR · 0.858
Following the CapMAS dual protocol: precision is the fraction of a caption's atomic propositions a multimodal judge finds supported by the image and the IIW reference description; recall is the fraction of human-verified multiple-choice questions answerable from the caption alone, with the image withheld. CLAIR is a reference-based LLM rating. Training uses MS COCO images only — the captions are discarded.
| Method | Model / signal | CLAIR | Precision | Recall | F1 | Length (words) |
|---|---|---|---|---|---|---|
| Inference-time baselines | ||||||
| Plain | Qwen2.5-VL-7B | 0.842 | 0.7884 | 0.6002 | 0.6815 | 347.55 ± 46.82 |
| CapMAS | 5-stage pipeline | 0.854 | 0.8467 | 0.6003 | 0.7025 | 189.71 ± 49.03 |
| PAPO baselines | ||||||
| PAPO | Qwen2.5-7B | 0.843 | 0.7905 | 0.5950 | 0.6790 | 181.86 ± 30.47 |
| PAPO + YOLO | Qwen2.5-7B | 0.842 | 0.7936 | 0.5969 | 0.6813 | 182.41 ± 32.71 |
| Reward-optimized baselines | ||||||
| FeedQuill | Qwen2.5-VL-7B | 0.833 | 0.7966 | 0.5967 | 0.6823 | 164.15 ± 69.48 |
| Ours | ||||||
| SimLoss FFT | Qwen3-VL-Embed | 0.844 | 0.8485 | 0.5991 | 0.7023 | 114.86 ± 13.06 |
| SimLoss GRPO | Gemini 2 Embed | 0.858 | 0.8227 | 0.6015 | 0.6949 | 132.87 ± 34.02 |
| SimLoss-PAPO-YOLO | Qwen2.5-7B | 0.855 | 0.8340 | 0.5970 | 0.6959 | 149.84 ± 29.56 |
Bold = best per column, underlined = second best. Length is mean ± standard deviation.
Across eight IIW-400 examples, SimLoss more consistently describes scene depth, paired objects, visible text, and fine-grained materials, with less repetition. Pick an image to read the SimLoss FFT caption and what to notice — including one honest failure case.
SimLoss FFT · 122 wordsThis image captures a serene and majestic mountain landscape. In the foreground, there is a wooden bench placed on a grassy area, inviting viewers to sit and enjoy the view. The bench is simple in design, with a backrest and seat made of natural wood, blending harmoniously with the natural surroundings. The middle ground is dominated by a sea of clouds that blanket the lower slopes of the mountains, creating a misty and ethereal atmosphere. The clouds appear thick and low-lying, partially obscuring the lower parts of the mountains and adding a sense of depth and mystery to the scene. In the background, the mountains rise majestically, their peaks sharp and jagged, with some areas covered in snow, indicating higher altitudes. …
SimLoss FFT · 132 wordsThis image captures a serene forest scene during what appears to be the spring season, as indicated by the abundance of white flowers blooming on the trees. The foreground features a large tree with a thick trunk and numerous branches, heavily laden with white blossoms. The branches extend outward, creating a dense canopy of flowers that dominate the view. The ground is covered with a mix of green grass and fallen leaves, suggesting a transition between seasons. In the background, there are other trees with a mix of green foliage and some bare branches, indicating that they might not have fully bloomed yet or are in different stages of growth. The sky above is clear and blue, adding to the overall tranquility of the scene. …
SimLoss FFT · 108 wordsThe image shows a train platform with a series of car carriers loaded with luxury cars. The train is labeled “MILSPED AML,” which likely stands for a company that specializes in the transportation of vehicles by rail. The cars are neatly arranged on the carriers, with some covered in protective plastic to prevent damage during transit. The platform has a modern design with a checkered pattern on the edge, and the sky above is clear with a few scattered clouds. The train is stationary, and the platform appears to be empty except for the cars. The overall scene suggests a professional and organized process for transporting high-value vehicles.
SimLoss FFT · 133 wordsThe image shows a section of a tiled wall with two mosaic designs of hats. The tiles above the hats feature a number “23” in a mosaic pattern, with blue and white tiles forming the digits. Below the number, there is a row of smaller, multicolored tiles that create a border around the number. The two hats are depicted on separate plates. The hat on the left plate is yellow with a red band and appears to have a red object, possibly a flower or a decorative item, placed on top. The hat on the right plate is black with a red band and also has a red object on top, which looks similar to the one on the yellow hat. …
SimLoss FFT · 112 wordsThe image depicts a mannequin dressed in medieval armor and attire. The mannequin is wearing a chainmail hauberk, which is a type of armor made of interlinked metal rings. Over the hauberk, there is a red surcoat, which is a loose-fitting garment often worn over armor for protection and visibility. The surcoat is secured with a white sash tied around the waist. The mannequin also wears chainmail gauntlets on the arms and chainmail boots that cover the legs up to the knees. A helmet with a visor is worn on the head, providing head protection. The mannequin is holding a sword in its right hand, which is sheathed at the hilt. …
SimLoss FFT · 113 wordsThe image shows a close-up view of a decaying log. The log is covered with a layer of pine needles and small pine cones, indicating it is part of a forest floor. The wood appears weathered and broken, with a dark, hollow center that suggests it has been hollowed out by decay or possibly by an animal. The texture of the wood is rough and uneven, with cracks and splits visible throughout. The pine needles and cones are scattered around the log, adding to the natural debris on the forest floor. The overall appearance of the log gives a sense of age and decomposition, typical of a fallen tree in a forest environment.
SimLoss FFT · 130 wordsThe image shows three cars parked on the side of a street in a residential area. The car on the left is a red Porsche, which appears to be an older model with a classic design. The middle car is a white SUV, which looks more modern and is parked behind the Porsche. The car on the right is a silver vintage car, possibly a classic Porsche as well, given its shape and design. The cars are parked in front of a house with a light-colored exterior and a tree in the background. The street is lined with trees and bushes, and there is a fence partially visible on the left side of the image. The overall scene suggests a quiet neighborhood with a mix of classic and modern vehicles.
SimLoss FFT · 121 wordsThe image depicts a large, intricately carved wooden sculpture of a deer. The sculpture is highly detailed, with the antlers and body of the deer crafted from wood that has been carefully shaped to mimic the natural form of the animal. The antlers are particularly elaborate, with multiple branches and points, giving the sculpture a dynamic and lifelike appearance. The deer's head is turned slightly to the side, and its mouth appears to be open, as if it is mid-roar or mid-breathe. The sculpture is mounted on a platform, which is part of an indoor setting, likely a museum or gallery, as indicated by the “WELCOME” sign in the background. …
On the two images below, Plain VLM, CapMAS, and SimLoss FFT answer an identical set of evaluation questions — coverage is exactly equal — yet caption length varies by 3×. The baselines' surplus words are summaries, speculation, and imposed structure, and they coincide with lower precision. Conciseness is a byproduct of optimizing for information content: the most efficient way to keep a caption discriminative is to fill it with grounded detail and little else.
“The image depicts a wooden sign mounted on a weathered wooden post. The sign is rectangular and appears to be handcrafted, with a white background and black text. The text on the sign reads: ‘All weapons must be peace tied….’ The word ‘peace’ is emphasized by being written in a larger, bolder font compared to the rest of the text. To the right of the text, there is a simple illustration of a knife. The knife is depicted with a wooden handle and a metallic blade. The illustration is minimalistic, with the handle being brown and the blade being gray. The knife is positioned horizontally, with the blade pointing to the right. Below the sign, there is a wooden box or container. The box is rectangular and made of light-colored wood, possibly pine or another softwood. The box appears to be empty and is mounted on the same wooden post as the sign. The box is positioned directly beneath the sign, suggesting that it might be used to store or organize items related to the sign's message. The background of the image shows the wooden post, which is weathered and has a rough texture. The wood appears to be old and has a natural, rustic appearance. The post is vertical and supports both the sign and the box. The overall setting suggests a rustic or outdoor environment, possibly a campsite, a cabin, or a similar location where the sign is meant to be visible and easily understood by visitors or residents. The sign's message, ‘All weapons must be peace tied….’, implies a rule or guideline for the safekeeping of weapons, possibly indicating that weapons should be secured or tied in a way that ensures they are not accessible or dangerous. In summary, the image shows a handcrafted wooden sign with a message about the safekeeping of weapons, accompanied by a simple illustration of a knife and a wooden box mounted on a weathered wooden post. The setting appears to be rustic and outdoor, likely in a campsite or similar environment.”
“The image depicts a rustic, weathered wooden post supporting a rectangular white sign with black text and a black-and-white illustration of a knife. The sign reads, ‘All weapons must be peace tied….’ The illustration shows a knife with a wooden handle and a metal blade. Below the sign, a light-colored wooden box or container is mounted on the same post, with a flat top and a front panel that appears slightly recessed. The box has a simple design and is likely used for storing or displaying items related to the sign's message. The wooden post is old and weathered, with visible signs of aging such as cracks, knots, and discoloration. The post is dark brown, indicating exposure to the elements for a long time. The overall scene suggests a rustic or outdoor setting.”
“The image shows a wooden sign attached to a wooden door or panel. The sign is white with black text and features an illustration of a knife. The text on the sign reads, ‘All weapons must be peace tied….’ The knife depicted on the sign has a wooden handle and a metallic blade. Below the sign, there is a piece of wood that appears to be part of the door or panel, and it is secured with some black zip ties. The background consists of weathered wooden planks, giving the sign a rustic appearance. The overall tone of the sign suggests a humorous or ironic message about the use of weapons, possibly indicating a playful or lighthearted context rather than a serious one.”
Plain VLM spends nearly a third of its words on a closing summary that repeats what was already said, plus speculation about the setting (“possibly a campsite, a cabin”). CapMAS infers the box's intended use without evidence. SimLoss FFT covers the same questions in 123 words at 100% measured precision against Plain VLM's 90.9%.
“The image depicts a circular stone carving or engraving on a textured surface. The carving is intricate and appears to be a stylized representation of a face or a symbol. The face is composed of several elements that are artistically integrated into the circular design. 1. Circular Outline: The entire design is contained within a circular boundary, which is the most prominent feature. The circle is not perfectly round but has a slightly irregular shape, giving it a naturalistic feel. 2. Central Face: At the center of the circle, there is a stylized face. The face is composed of several elements: Eyes: Two small, round eyes are positioned symmetrically on either side of the face. The eyes are simple, with small dots representing the pupils. Nose: A small, round nose is located between the eyes. Mouth: A simple, curved line represents the mouth, which is smiling. Hair: The hair is depicted with several curved lines and loops, giving it a flowing and dynamic appearance. The hair extends outward from the face, creating a sense of movement and life. 3. Additional Elements: Waves and Curves: The carving includes various waves and curves that flow around the face, adding to the dynamic and fluid nature of the design. These elements are integrated into the circular boundary, enhancing the overall aesthetic. Symmetry: The design is symmetrical, with the left and right sides of the face mirroring each other, contributing to a balanced and harmonious appearance. 4. Texture and Surface: The carving is set against a rough, textured background, which appears to be a stone or concrete surface. 5. Artistic Style: The style of the carving is reminiscent of ancient or traditional art forms, possibly inspired by symbols or faces found in historical or cultural contexts. The simplicity and elegance of the design suggest a focus on essential elements and a strong emphasis on symmetry and balance. In summary, the image shows a detailed stone carving of a stylized face or symbol, set against a textured background. The design is intricate, with elements such as eyes, nose, mouth, and flowing hair, all integrated into a circular boundary. The overall style is symmetrical and balanced, with a focus on simplicity and elegance.”
“The image depicts a circular stone carving or etching on a textured, light-colored surface. The carving features two intertwined figures, which appear to be stylized representations of human-like forms, with simplified facial features, including eyes, noses, and mouths. The figures are depicted in a flowing, interconnected manner, creating a sense of unity and harmony. The central part of the carving features two faces, each with eyes represented by small circles. The figures are adorned with intricate patterns and designs, including curved lines that form a pattern resembling a crown or a series of waves. These patterns add a decorative element to the carving. The background of the carving is a textured stone surface, which appears to be weathered and slightly rough, adding depth and contrast to the carving. The stone's color is a muted gray, which complements the darker lines of the carving, making the design more pronounced. The circular nature of the carving suggests a sense of unity and wholeness, as the two figures are seamlessly connected within the same space. The intertwined forms could symbolize various concepts such as unity, balance, or the interconnectedness of life. The simplicity of the design, combined with the rough texture of the stone, gives the carving a timeless and enduring quality. The stone surface on which the carving is etched has a rough texture, with visible grains and imperfections, adding a rustic and natural feel to the artwork. The contrast between the smooth lines of the carving and the rough texture of the stone surface creates a visually interesting and dynamic effect. The image presents a harmonious and artistic stone carving that combines abstract human figures with decorative patterns, set against a textured stone background. The design focuses on simplicity and elegance, making it a powerful and meaningful piece of art.”
“The image shows an intricate circular design etched into a textured surface, likely concrete or stone. The design features two intertwined, stylized faces with exaggerated, flowing lines that give them a dynamic and almost animated appearance. Each face has a large, central eye and a smaller eye on the side, both depicted with simple, curved lines. The eyes are surrounded by elaborate, swirling patterns that extend outward, creating a sense of movement and fluidity. The overall effect is reminiscent of ancient or tribal art, with a focus on symmetry and organic forms. The texture of the surface adds depth to the design, making it appear three-dimensional.”
Plain VLM structures its 395 words as a numbered breakdown with headers and a full closing summary, none of which adds answerable content. CapMAS interprets the figures as symbols of “unity, balance, or the interconnectedness of life” — unverifiable from the image. SimLoss FFT answers the same questions in 106 words at 100% precision against 94.1% (Plain) and 89.7% (CapMAS).
It aligns the VLM's continuous hidden state with a frozen image embedding — a dense signal applied before text generation. Unlike caption imitation, it never copies a pseudo-reference's wording or omissions; unlike RL, it avoids high-variance updates from sampled text. The model is pushed to preserve image-specific evidence in its internal representation.
The black-box reward — cosine similarity between caption and image embeddings — encourages broader coverage (best recall, 0.6015). But because it is applied after discrete generation, the model can also raise similarity by adding visually plausible content, which costs precision relative to FFT.
Explicit decompose–verify–rewrite is well suited to suppressing false positives. But the margin is 0.0002 F1 — bought with a five-stage pipeline at 115.31 s/image against 5.77 s. SimLoss FFT occupies the more favorable quality–latency point.
CapMAS filters propositions with a multimodal judge; FeedQuill rewards captions with one. LLM judges show a systematic agreeableness bias — accepting valid outputs at ~96% but flagging invalid ones below 25% — missing exactly the plausible hallucinations that matter. SimLoss's supervision is a frozen image embedding, not a judge verdict.
SimLoss trades dependence on caption targets for dependence on a frozen embedding teacher: its biases and visual granularity determine which distinctions are rewarded — and greater specificity can amplify fine-category errors (the driftwood moose). The remaining gap to CapMAS suggests explicit verification still helps precision–recall balancing. Natural next steps: comparing or ensembling embedding teachers across domains, combining FFT's precision with GRPO's recall, and lightweight factuality checks that preserve the single-pass deployment path.
@misc{jain2026glanceneedsinglepassfinegrained,
title={A Glance Is All You Need: Single-Pass Fine-Grained Image Captioning with SimLoss},
author={Suryaansh Jain and Rahasya Barkur and Vishal G and Ryan Rossi and Franck Dernoncourt and Jack Wang and Koustava Goswami and Nedim Lipka and Puneet Mathur and Samyadeep Basu and Seunghyun Yoon},
year={2026},
eprint={2609.00591},
archivePrefix={arXiv},
primaryClass={cs.CV},
url={https://arxiv.org/abs/2609.00591},
}