1. During Stage 1 of training, we follow the pipeline established by Stable Diffusion 2.1 base. Specifically, we first obtain conditioning features from the VLV encoder. These features are then passed through our trainable projection layer followed by the CLIP Text Encoder from SD 2.1. The resulting projected conditions, together with random noise, are input into the U-Net from SD 2.1 to predict the noise residuals. We employ a standard mean squared error (MSE) loss for supervision in this stage. Throughout training, we adopt the default 1000 denoising steps used in SD 2.1 for the noise prediction process. For inference (image generation), we follow the same conditioning pipeline and generate images using the SD 2.1 sampling procedure with 50 denoising steps.

2. It is indeed true that the VLMs are sensitive to the prompt template. Thus, for fair comparison, we use the same prompts for VLMs. Specifically, for LLaVA-v1.5-7B and Qwen-2.5-VL-7B, we use the same prompts. For Florence-2 Large, it has a fixed prompt template; for fair comparison, we use <MORE_DETAILED_CAPTION>. Please refer to the table below for detailed caption sources. | Model | Prompt / Caption Source | |----------------------|----------------------------------------------------------------------------------------------| | Original | Captions from MS-COCO Dataset | | Recap-7B | Captions from [1] . | | LLaVA-v1.5-7B | Please describe this image as detailed as possible. | | Qwen-2.5-VL-7B | Please describe this image as detailed as possible. | | Florence-2 Large | <MORE_DETAILED_CAPTION> | | VLV | No prompt | Table: Prompt templates or caption sources used for each model.

3. For Table 6, we randomly select 200 images from laion2B-en-aesthetic as the validation set. And we prompt Gemini 2.0 Flash to predict the 3D bounding box for those images and then we visualize the boxes by ourselves. For Section 4.4, Emerging Properties, the “emerging” aspect we highlight refers to the fact that our training pipeline unexpectedly yields images whose spatial structure can be reliably recovered in 3D—even though the model was never explicitly trained for that task.

4. Configuration of Transformer encoder-decoder:
   The model employs a symmetric, sequence-to-sequence Transformer with 12 encoder and 12 decoder layers, each with a 1024-dimensional hidden size. Every layer uses 16-head self-attention, followed by a feed-forward network that expands to 4096 dimensions and applies a GELU activation. Regularization mirrors Stable Diffusion defaults: a 0.1 dropout is applied to activations, attention, and residual connections. The tokenizer supports sequences up to 4 096 tokens and uses the standard identifiers BOS = 0, PAD = 1, and EOS = 2. All weights are initialized from a normal distribution with σ = 0.02, ensuring stable training.
   Configuration of MLP in VLV Encoder:
   The MLP in the VLV Encoder is implemented as a two-layer feedforward network with a GELU activation in between. It projects language features from a 1024-dimensional input to a configurable hidden dimension (hidden_dim) and then back to 1024 dimensions.

5. We agree with the reviewer that handling OCR-related content is an important aspect for captioning models. As noted in our limitations section, our current model is not well-suited for OCR tasks. This limitation primarily arises from Stable Diffusion 2.1 itself. Stable Diffusion 2.1 is poor at generating images with plenty of words on those. Thus, we cannot distill the ability of capturing the OCR-related content from SD 2.1.

6. It is true that SD2.1 is a bit old; due to the computational resources, it is hard for us to apply SD3.5 and FLUX. Applying state-of-the-art diffusion decoders is also our future work.

Minors:

1. We follow this setting from De-Diffusion[2] work and Adafactor[3].
2. Sorry for the confusion, we perform our experiment with both BF16 and FP16, and we find BF16 is the better option.

Reference:
[1] What If We Recaption Billions of Web Images with LLaMA-3?
[2] De-Diffusion Makes Text a Strong Cross-Modal Interface
[3] Adafactor: Adaptive learning rates with sublinear memory cost

Thanks for the rebuttal. As most of my concerns have been addressed, I decide to raise my score. It's better to include modern t2i models to make this paper stronger.

1. To clarify our contribution: the core representation-learning stage relies exclusively on unpaired images, enabling the model to distill language-semantic features directly from visual data. Captioned text–image pairs are introduced only in a subsequent alignment phase to map the already-learned visual latent space onto the textual domain; they do not provide supervision for learning the underlying representations themselves.

2. De-Diffusion is not open-sourced, so a direct, fully controlled comparison is impossible. With their authors’ cooperation, we attempted to reproduce their pipeline; however, convergence proved unstable because their discrete-token approach relies on a Gumbel-Softmax relaxation whose temperature schedule is undocumented. In contrast, our method operates with continuous text embeddings, making the entire image-to-text pathway differentiable end-to-end and eliminating the need for potentially brittle discrete sampling. Moreover, we build on openly available pre-trained models and public datasets, ensuring that our codebase is reproducible for the research community and readily scalable in industrial settings. We believe these practical and methodological advantages provide a fair—and, for many applications, more deployable—alternative to De-Diffusion.

3. We added some experimental settings as Reviewer CiwQ requested, we hope those clarifications could help you have a better understanding of our work.

4. As requested, we have added a LoRA-based baseline to Stage 2. Specifically, we fine-tuned Qwen-2.5 (3 B) with LoRA rank = 16 under exactly the same hyper-parameter settings used for the full-parameter baseline. After training, we generated captions for the same 30 K COCO validation images and assessed quality by prompting SD-3.5-medium and computing FID with guidance scale = 2. The LoRA variant reduces GPU memory consumption substantially (only the low-rank adapters are updated), but, as shown in Table here, this comes at a noticeable cost in caption quality, with FID increasing relative to full fine-tuning. These results confirm that while LoRA is attractive for resource-constrained scenarios, full fine-tuning remains preferable when maximum performance is required. | Method | FID ↓ (GS = 2) | Peak GPU Memory (GB) per GPU | |----------------------|----------------|------------------------------| | LoRA (rank 16) | 8.09 | 20 GB | | Full fine-tuning | 6.64 | 47 GB |

5. Thank you for pointing out the ambiguity regarding the 3D bounding boxes in Section 4.4 (“Emerging Properties”). To clarify: the 3D bounding boxes are not produced directly from the text embeddings. Instead, after generating images from the text embeddings, we prompt Gemini 2.0 Flash to extract their 3D bounding boxes. The “emerging” aspect we highlight refers to the fact that our training pipeline unexpectedly yields images whose spatial structure can be reliably recovered in 3D—even though the model was never explicitly trained for that task. We will update the manuscript accordingly to make this workflow clear and avoid further confusion.

Dear Reviewer pgRD,

Thanks for your contribution to NeurIPS, and thanks for submitting your Mandatory Acknowledgement!

Please note that the deadline of Author-Reviewer Discussions is fastly approaching (July 31 - Aug 6). Please acknowledge the authors' rebuttal as soon as possible.

Thanks

AC

1. We report FID rather than text-only metrics such as CIDEr because our task is detailed captioning: a caption is good if it contains enough information for a diffusion model to recreate an image that is perceptually close to the original. Concretely, we feed each caption to Stable Diffusion 3.5-Medium, which accepts prompts longer than 77 tokens, generate an image with identical denoising steps, resolution, guidance scale, and sampler across all experiments, and then compute the Fréchet Inception Distance between the generated and source images; since every setting except the caption is fixed, lower FID directly implies a richer, more accurate description. By contrast, CIDEr, BLEU, and SPICE depend on ground-truth references whose average length on MS-COCO is only about ten words, so they systematically penalise the longer, fine-grained sentences produced by modern VLMs. While FID thus provides a fairer gauge of descriptive adequacy in our scenario, we agree that the community still lacks a universally accepted metric for high-detail captioning.
2. Sorry for the confusion on the base model we use in our main experiments. We use Qwen2.5-3B as our LLM decoder in our training stage 2. Qwen 2.5-3B is the largest model we could afford given our GPU budget for fully fine-tuning. But we do perform scalability experiments with Qwen2.5-0.5B, Qwen 2.5-1.5B, and Qwen2.5-3B. From the table here, it is obvious that with the increase of the LLM decoder size, the performance on the image captioning task improves. And according to the experiment here, we can infer that we can get better results with Qwen2.5 -7B or more powerful LLM. | Decoder | GS=1 | GS=2 | GS=3 | GS=4 | |----------------------|---------------:|---------------:|---------------:|---------------:| | Qwen-2.5-0.5 B | 14.70 | 9.37 | 11.26 | 12.45 | | Qwen-2.5-1.5 B | 12.25 | 7.30 | 9.16 | 10.26 | | Qwen-2.5-3 B | 11.47 | 6.64 | 8.56 | 9.90 |
3. Given our computational constraints, it was infeasible to adopt heavier diffusion back-bones such as SD-3.5, or FLUX for Stage 1, so we relied on Stable Diffusion 2.1. As illustrated in Fig. 3 (second row), SD-2.1 effectively defines our performance ceiling: when we reconstruct images directly from our caption embeddings, minor detail mismatches still appear, indicating that the decoder—not the caption model—is the limiting factor. To narrow this gap in Stage 2 we unfreeze the VLV encoder and train it jointly with the auto-regressive decoder; although this choice modestly increases memory usage, it yields clearly more faithful captions (Table 4). We therefore conclude that, resources permitting, pairing our pipeline with a more powerful diffusion model would further boost caption fidelity, while the current results already represent the best achievable quality under SD-2.1.
4. Thank you for pointing this out. Our intent in Lines 240–241 was not to claim that low absolute cost alone proves scalability, but rather to highlight that our pipeline already matches state-of-the-art performance despite running with modest-capacity diffusion and language decoders. The ablations in Table above show that when we increase only the LLM decoder from 0.5 B → 3 B parameters, caption quality (FID↓) improves monotonically. Thus, we expect better performance with a powerful diffusion decoder and LLM decoder. We will make this statement clearer in our manuscript once it is accepted.

1. The previous works, like LLaVA have already done this. And in our Table1, we have proved that our pipeline is better than previous ones. Moreover, compared with our pipeline, the CLIP requires high-quality text-image pairs. While in our pipeline, we can learn language semantics representation via images only in our training stage 1 by establishing an information bottleneck.
2. Retraining the entire system from scratch is unnecessary when moving to newer SD versions. In Stage 1, the VLV encoder is trained with an image-only, information-bottleneck objective and never encounters CLIP tokens. To accommodate a T5 text encoder, we simply introduce a lightweight linear/MLP adapter and fine-tune it. Concretely, we trained a two-layer MLP with a ReLU activation to map CLIP text embeddings to the T5 embedding space; after 2 000 steps on 512 k captions, the validation loss dropped to 0.003. Replacing the T5 encoder in SD 3.5 with the CLIP encoder from SD 2.1 plus this trained adapter still produced correct images. We use this pipeline for T2I tasks. Specifically, we generate hundreds of images with the corresponding captions and calculate the CLIP scores. We get the mean a CLIP score of 28.88.
   Because the text encoder in our pipeline remains frozen, functionally acting as a “large MLP layer”, adapting to any new text space only requires adding and briefly tuning a similarly small adapter.
3. To endow our model with genuine image understanding—i.e., the ability to map raw pixels to rich, language-semantic representations—we build the VLV encoder on DaViT, following Florence-2[1]. As Tables 7 and 9 in Florence-2 show(we put the table below for reference), DaViT consistently surpasses alternative backbones (ViT variants, Swin, ConvNeXt) on detection and segmentation benchmarks, indicating superior capacity for high-level visual reasoning. Leveraging this stronger visual core allows the VLV encoder to comprehend scene layout, object attributes, and relational cues more effectively. Consequently, substituting DaViT with other backbones would be unlikely to improve results within the same pipeline.

Mask R-CNN / DINO Performance

ADE20K Semantic Segmentation (UperNet)

4. We acknowledge that feeding our caption embeddings into Stable Diffusion 2.1 cannot reproduce fine-grained visual details—the generation quality is capped by SD 2.1 itself. To mitigate this ceiling, Stage 2 trains an auto-regressive caption decoder jointly with a trainable VLV encoder. Making the visual encoder update alongside the decoder allows it to refine its representations toward the specific descriptive cues the decoder needs, yielding noticeably richer captions. The ablation in section 4.3 (Table 4) confirms the benefit: enabling VLV encoder updates yields the best performance among all other settings, despite the generative ceiling imposed by SD 2.1.

5. Yes. We use laion2B-en-aesthetic for stage 1 training to learn the language-semantic representation. This dataset is a subset of LAION-5B, which indeed contains stylized images, such as birthday greeting cards or posters. However, we filter out those images due to our criteria shown in the supplementary materials, Section A, data processing. We also explain that our model right now is not good at OCR due to the ability of SD 2.1. But our model could be applied to specific scenarios, fine-tuned on a corresponding dataset.

Reference:
[1] Florence-2: Advancing a Unified Representation for a Variety of Vision Tasks